Fix consensus: use modal partition as initial_membership on original edges
Co-occurrence edge-scaling caused over-fragmentation (260-282 clusters vs ~214)
because lower edge weights reduce modularity at fixed resolution=5.0.
Modal initialization assigns each node to its most-frequent cluster across 25
restarts, then runs Leiden on original (unscaled) edges from that starting point.
Preserves the calibrated modularity landscape; boundary-oscillating contigs are
pre-separated by majority vote before Leiden refines.
cb0de7d
Fix consensus Leiden: apply to seed sweep path (winning config)
The winning config (--leiden-restarts 25 without --quality-leiden) uses the
seed sweep path (skip_phase2=true), not run_adaptive_restarts. The previous
consensus code was in the wrong branch and never executed.
Move consensus to the seed sweep path: after N=25 restarts, build
co-occurrence edges w_consensus = raw_w * (co_count / N), then run 3 final
Leiden seeds on the consensus graph and take best of all.
9f16968
Add consensus Leiden from restart co-occurrence matrix
After the 3-stage restart search, collect all P1 partitions and build
consensus edge weights: w_consensus = pow(raw_w, bw_orig/best_bw) * co_frac
where co_frac = fraction of restarts where nodes u,v were co-clustered.
Partition-boundary contigs (contaminating genomes that oscillate between the
primary bin and the garbage cluster across seeds) get co_frac << 1 with the
primary genome → their edges are down-weighted → Leiden reliably finds the
correct cut regardless of seed, converting 11 HQ variance to stable 12 HQ.
Zero extra compute: uses the 25 partitions we already collect, then 3 final
Leiden runs on the consensus graph. Takes best of consensus and best restart.
9f9ffa4
Add Phase 4E: extended-neighborhood Leiden re-clustering
Phase 4A runs Leiden on the isolated bin subgraph — it has no external edges,
so it cannot see that contaminating contigs are kNN-connected to other clusters.
This is why Phase 4A gets 0 splits on near-HQ contaminated bins.
Phase 4E fixes this by including 1-hop kNN neighbors from other bins in the
subgraph. External neighbors anchor contaminating contigs to their true genome
cluster, making the partition boundary visible to Leiden. This is the mechanism
behind the 12 HQ vs 11 HQ variance: different Leiden seeds find (or miss) this
same cross-cluster cut.
Tries 5 seeds × 5 resolutions per contaminated bin. Accepts if any gives a
clean sub-group (cont < 5%, comp loss ≤ 5%), with sole-carrier promotion.
207656f
Add Phase 4G: cross-cluster graph-boundary eviction
Contaminating contigs sit at the kNN boundary between the primary genome
cluster and other clusters — the 12 HQ vs 11 HQ variance arises when
Leiden's random seed moves these boundary contigs to the garbage cluster
instead of the primary genome bin.
Phase 4G makes this deterministic: rank contigs by
cross_ratio = w_out / (w_in + w_out)
using the full adj_ graph, greedily evict highest-ratio contigs until
CheckM cont < 5% (sole-carrier + comp-loss ≤ 5% guards).
Runs after Phase 4M (marker-direct), before Phase 4C (curvature).
1d7e6af
Add Phase 4C: Gaussian curvature decontamination via local SVD
For each contaminated bin (cont > 5.5%), compute per-contig off-tangent-plane
score using SVD of the k-NN embedding neighborhood. Contaminating contigs sit
at the boundary between two genome manifolds → their k-NN spans both clusters
→ high off-tangent residual (Gaussian curvature proxy).
Greedy eviction: add highest-curvature contigs until CheckM cont < 5%, with
≤5% completeness loss and sole-carrier protection. Catches contamination that
Phase 4M misses (no duplicate SCG markers on the contaminant).
Uses existing 157-dim embeddings (already computed), no new parameters.
5a0ab9b
Fix Phase 4M: sole-carrier protection + strict acceptance
- Filter out eviction candidates that are the sole carrier of any marker
in the bin — evicting these destroys completeness
- Replace loose acceptance (chimeric/near-HQ split) with single criterion:
remaining cont < 5% AND comp loss ≤ 5%
- Ensures Phase 4M only cleans bins that actually become HQ-clean
6af7408
Add Phase 4M: marker-direct contig eviction for decontamination
For each contaminated bin after Phase 4A, find duplicate SCG marker
carriers, evict the lowest intra-bin connectivity contig per duplicated
marker, validate with CheckM. No threshold tuning — SCG identity
directly measures what causes contamination.
503bf1e
Revert Phase 4B TNF decontamination (0 splits, threshold-sensitive, no benefit)
116383e
Phase 4B: more aggressive TNF outlier detection (0.7-0.8 factor, 0.92 sim threshold, 10% chimeric floor)
a589cbc
Add Phase 4B: TNF signature-based decontamination
After graph-based Phase 4A Leiden splitting, run TNF cosine-similarity
connectivity analysis on remaining contaminated bins. Identify outlier
contigs (low core similarity = contaminating genome), split into core
+ outlier groups, validate with same CheckM dual acceptance criterion.
Uses tnf_graph::refine_bin_tnf() (existing header-only algorithm).
outlier_factor: 0.55 for cont>=15%, 0.45 for cont 5-15%.
Both halves must be >= restart_min_viable_bp to accept.
2f153d2
Phase 4: remove completeness floor, dual acceptance (chimeric vs near-HQ), iterative splitting
87e0fb3
Phase 2/4 resolution sweep: try 0.05x-3x to split large contaminated bins
0e760ae
Fix decontaminate() access: move to public section
6d0078b
Move Phase 4 decontamination to main pipeline, runs for all restart configs
f9cdd00
Fix SWA BN recalibration: use cumulative momentum (momentum=None) for exact running stats
4c0c2de
Add targeted Phase 4 decontamination for HQ-completeness contaminated bins
Adds min_completeness parameter to run_phase2() (default 0 = all bins,
unchanged). After Phase 3 rescue, unconditionally runs a targeted Phase 2
pass restricted to bins >= 85% CheckM completeness AND > 5.5% contamination.
This avoids skip_phase2 restriction: the plain seed sweep disables global
Phase 2 to prevent over-fragmentation of incomplete bins, but HQ-completeness
bins that are contaminated still benefit from splitting. Phase 4 catches these
cases (e.g. bin_10: 93.5%/7.4% consistent across all reps).
Accepts splits only via the existing CheckM criterion: child_hq > parent_hq
OR weighted-average contamination drops >= 1% within 2% completeness.
b3b3d55
Fix SWA: recalibrate BatchNorm running stats after weight averaging
After apply_swa(), BatchNorm buffers (running_mean/var) remain calibrated
for the last epoch's non-averaged model. The averaged parameters shift the
activation distribution, making BN statistics stale.
Fix: reset running stats and do a forward pass over all training batches
in train mode (no grad) immediately after averaging. Applies to both the
standard and damage-aware train_multiview paths.
1d083e2
Add 3 levers to push HQ count past 11 consistently
Lever 1 - scg_score_labels() scoring tiers:
- Add num_strict_hq (90%/5% internal, ~CheckM2 100%) at 1e8
- Add num_pre_hq (72%/5% internal, ~CheckM2 87%) at 5e5
- Provides gradient both above and just below the 75% HQ threshold
- Restarts now prefer partitions approaching the CheckM2 90% boundary
Lever 2 - Phase 3 targeted SCG rescue:
- After Phase 2 / Phase 1b, identify near-HQ bins (75-90% internal, <5% cont)
- For each target, scan kNN neighbors for marker-bearing contigs whose
markers are ALL absent from the bin (safe to recruit, no dup risk)
- Donor protection: don't steal from large viable bins unless removing contamination
- Directly pushes borderline bins over the completeness threshold
Lever 3 - SCG complementary edge bonus:
- Add scg_complement_bonus = 0.3f to QualityLeidenConfig
- In both the restart and single-run edge weighting paths, boost edges
between marker-bearing contigs with non-overlapping marker sets by +30%
- Complementary marker pairs (same genome, different contigs) are stronger
in embedding space; bonus helps Leiden keep them co-clustered
420913e
Refactor InfoNCE loss to logsumexp+gather, eliminate masked_select
Replace the COMEBin-style [positive, negatives] construction with a
logsumexp formulation that avoids all masked_select calls on the N×N
similarity matrix:
loss_i = logsumexp(sim[i]) - mean(sim[i, positives])
This eliminates the largest per-batch bottleneck (scatter/gather over
6144×6144 = 37.7M elements per batch).
Damage weights now applied via block-broadcast instead of repeat():
sim.view(nv, bs, nv, bs).mul_(cw_gpu.view(1, bs, 1, bs))
This removes the 144MB cw_tiled allocation (cw_gpu.repeat({6,6})).
Cache pos_idx_ (N×(nv-1) positive-index tensor) as a member — computed
once per (bs, nv) configuration, never again.
Remove compute_accuracy() free function; accuracy now computed directly
from sim.argmax() vs pos_idx_.
DamageAwareInfoNCE::forward() replaced with compute_weights() which
returns the (bs, bs) weight matrix; all loss logic in AmberTrainer.
fd813ea
Fix training bottlenecks: 36x damage weight reduction + remove per-batch GPU syncs
Damage weight matrix (damage_aware_infonce.h):
- Was: 6144x6143 CPU loop (37.7M calls) + 150MB host->device copy per batch
- Now: 1024x1024 CPU loop (1.05M calls) + 4MB copy, then tile on GPU with .repeat()
- Weight(view_i, view_j) = f(damage[i%bs], damage[j%bs]) — view-level matrix is just
a tiled repetition of the contig-level matrix; computing it 36x redundantly was wrong
GPU sync reduction (amber_encoder.h, all 3 training paths):
- Was: loss.item<float>() + compute_accuracy() called every batch (54 syncs/epoch)
- Now: accumulate loss as GPU tensor, single .item() call per epoch (2 syncs/epoch)
- compute_accuracy() only called on last batch (needed for early stopping check)
94115e1
Remove remaining COMEBin references from runtime and type names
- Fix consensus kNN log: 'base seed=42, hash-spread' instead of stale range
- Rename COMEBinScore → BinQualityScore, compute_comebin_score → compute_bin_quality_score
- Update runtime log messages: 'Training AMBER contrastive encoder', 'AMBER BAM loading', etc.
b0949ce
Resolve data paths relative to executable directory
AMBER was only usable from its own project directory because all data
file paths (HMM, marker sets) were hardcoded as relative paths.
find_amber_data() tries CWD first (backward-compat), then looks relative
to the executable parent directory (e.g., build/../auxiliary/). This lets
users run 'amber_gpu bin2 ...' from any working directory as long as
auxiliary/ and scripts/ are siblings of the build/ directory.
0adfd82
Fix ablation config 9: add --quality-leiden for adaptive BW restart search
Config 9 and 10 had identical flags (both plain seed sweeps). Config 9
is supposed to test adaptive bandwidth restart search, which requires
--quality-leiden. Config 10 remains the plain seed sweep at fixed bw=0.20.
Also rename config 9 to damage_infonce_cgr_bwrestarts to clarify.
9db7ee5
Use hash-spread seeds for Leiden and encoder restarts
Sequential base+i seeds have RNG correlation: nearby seeds produce
nearly identical initial Leiden node orderings and encoder weight
initializations. Replace with Knuth multiplicative hash (spread_seed)
which maps (base, counter) to well-distributed 31-bit values.
Affects three sites:
- quality_leiden_backend: next_seed lambda in run_adaptive_restarts
- quality_leiden_backend: plain seed sweep loop
- bin2.cpp: consensus kNN encoder restart loop
d35eea7
Add consensus kNN, SWA, oracle fix; rename COMEBin→AMBER
Key changes:
- ConsensusKnnBuilder: N encoder runs, aggregate kNN edges (freq≥2/3),
suppress brittle cross-genome bridges. Flag: --encoder-restarts N
- SWA in AmberTrainer: average weights over last 30 epochs (start=70)
- Oracle fix: scg_score_labels thresholds 90→75 (HQ), 70→50 (MQ),
50→30 (near-HQ) to match CheckM2 empirically; lexicographic scoring
- Rename COMEBinEncoder→AmberEncoder, COMEBinTrainer→AmberTrainer,
comebin_encoder.h→amber_encoder.h, compute_comebin_edges→compute_edges
- Ablation config 12: damage_infonce_consensus3 (--encoder-restarts 3)
56addf4
Add config 11: damage_infonce + seed sweep, no CGR
a7d3de9
Add pure seed sweep path for plain --leiden-restarts
When --leiden-restarts N is used without --quality-leiden:
- skip_phase2=true, marker_edge_penalty=0.0f (set in bin2.cpp)
- cluster() takes the do_restarts && skip_phase2 branch
- Runs N Leiden seeds at fixed original_bandwidth, no BW variation
- Selects best by lexicographic CheckM score (HQ primary)
- No marker edge penalization, no Phase 2 contamination splitting
Ablation script: add config 10 (damage_infonce_cgr_seeds25),
update header comment, fix bin2.cpp log message for plain restarts.
887f93c
Decouple --leiden-restarts from --quality-leiden
--leiden-restarts N now runs bandwidth sweep over plain Leiden without
marker edge penalization or Phase 2 contamination splitting.
Root cause of 10 vs 11 HQ regression: --leiden-restarts implicitly enabled
use_quality_leiden, activating marker_edge_penalty=3.0 which pushed 78
small contigs of a target genome (comp=90.2->82.6, cont=2.9->1.5) into
a junk cluster (bin_13, comp=9.4%).
Changes:
- cmd_bin2.cpp: remove implicit use_quality_leiden from --leiden-restarts
- bin2.cpp: build marker_index when n_leiden_restarts>1 (needed for SCG
scoring to guide bandwidth arm selection); create QualityLeidenBackend
in plain-restart mode with marker_edge_penalty=0, skip_phase2=true
- quality_leiden_backend.h: add skip_phase2 config field
- quality_leiden_backend.cpp: guard Phase 2 in evaluate_candidate() and
cluster() with skip_phase2 check
--quality-leiden --leiden-restarts N still combines both (unchanged).
8763075
Fix UCB bandit correctness bugs in bandwidth restart search
Three bugs found via OpenCode review:
1. Non-stationary P1/P2 score mixing in arm stats: arms that received P2
evaluations had inflated means, creating a feedback loop. Fix: track
bandit statistics (mean_p1, m2_p1, pulls_p1, best_p1) on P1 scores only.
P2 scores update global_best only.
2. do_p2 gate scale mismatch: compared selection_score() (may be CheckM
lexicographic, ~1e6 per HQ bin) against contamination_penalty (5.0),
making the gate always true once any P2 result existed. Fix: gate in P1
score space (score_p1 vs global_best_p1), where contamination_penalty
is a consistent ~1 SCG unit.
3. Stage 3 arm selection used mean instead of best: for best-of-budget
search with heavy-tailed Leiden stochasticity, mean picks the wrong arm.
Fix: select by best_p1 (ceiling, not mean).
Also document Phase 2 adjacency intent: uses original-bandwidth edges
intentionally — bandwidth trials affect cluster formation only.
eedec06