Skip to content

Torch-eCpG v2.0.0 Beta 1

Pre-release
Pre-release

Choose a tag to compare

View all tags
@kordk kordk released this 29 May 18:55
· 9 commits to main since this release
v2.0.0-beta.1
1435082

Torch-eCpG v2.0.0-beta.1 — Release Notes

Pre-release of the dev branch, accumulating ~18 months of work since Release 1.0.0 (__version__ = '0.0.1'). The 2.0 line graduates Torch-eCpG from a GPU MLR kernel into a full, reproducible eQTM analysis pipeline: scalable compute, end-to-end automation, rich downstream visualization, and a hardened annotation/QC layer.

Headline Additions

1. New MLR Backend & Bootstrap Inference

  • New torch.linalg.lstsq MLR backend with validation tests against the legacy path (1.1.0-dev).
  • lstsq bootstrap subcommand with a configurable bootstrap batch size and a new tools/createBootstrapList.py for filtering candidate eQTMs (1.7.0–1.7.1-dev).
  • Integrated Gradients (IG) integrated into the lstsq backend, with decoupled IG outputs and covariate filtering by file (1.6.0-dev, 1.6.5-dev).
  • Bit-equivalent inner-kernel peak-memory reduction in the lstsq path (in-place buffer assembly, early X free, K=1 slicing for methylation-only/no-IG; verified against pre-1.20.1 baseline) (1.20.1-dev).

2. Parquet I/O and Output Tooling

  • First-class Parquet output with Snappy/ZSTD compression in mergeOutputs.py (1.4.1-dev) and summarizeOutput_parquet.py for FDR on Parquet outputs (1.6.0-dev).
  • summaryParquetToCsv.py, recalculate_pvalues_parquet.py, and assignRegionToEcpg_parquet.py (GFF/GTF/BED6 + region assignment with verify_alignment) (1.5.0–1.6.3-dev).
  • Numerous Parquet hardening fixes — chunk-merge contamination at the 1,000,000-hit limit, pandas type inference for nullable int columns, gt_id/mt_id index-vs-column handling, sample_reservoir.csv exclusion in merges, and unique-count restoration in mergeOutputs.py (1.25.0–1.26.1-dev).

3. Host-Aware Performance & Memory Management

  • New --host-profile {auto,minimum,server} (env TECPG_HOST_PROFILE) drives defaults for save-pool size, output format (auto → Parquet on server, CSV on minimum), prefetch depth, and chunk auto-sizing (1.20.1-dev).
  • --prefetch-chunks with a ThreadPoolExecutor to overlap chunk preparation with GPU compute, plus a --blas-threads pre-import shim (1.13.0-dev).
  • Anchored auto-chunking: supply one of -g/-m and the other dimension is bisected against an 80%-of-budget memory target (CUDA mem_get_info / psutil) (1.21.0-dev).
  • IG-aware chunk estimator with bisection-based fallback and a VRAM safety ceiling that fixes a severe OOM regression on tight GPUs (L4-class, ~22 GB) when --compute-ig was set (1.22.2-dev).
  • Auto-scaled --save-threads, streaming to_csv, save-queue backpressure, switch to ThreadPoolExecutor for Parquet (GIL-released by PyArrow) while CSV keeps the ProcessPoolExecutor, and removal of per-chunk empty_cache() overhead (1.8.x–1.14.0-dev).
  • QR-decomposition cache and non-blocking NVML calls for higher GPU utilization (1.11.0-dev).
  • Rich per-chunk profiling and bottleneck heuristics (GPU/I/O/CPU-bound classification), GPU UUID-based monitoring, thermal/memory reporting (1.2.x, 1.12.x-dev).

4. End-to-End Pipeline Automation

  • New pipeline.sh orchestrating the full analysis with argument parsing, descriptive logging, timestamps, and data-existence checks (1.8.0-dev).
  • PCA covariate preprocessing integrated into the pipeline, with a standalone PCA script (1.8.0-dev).
  • pipeline.sh stage-resume via -s / --start-stage (e.g. pca, map, summarize) (1.27.0-dev).
  • New pipelinePost.sh automating downstream visualizations and network analyses (1.26.3-dev).
  • tools/exploreOmics.py integrated into the pipeline for QC of methylation/expression, with a consolidated HTML report comparing original vs. processed inputs (1.23.2-dev, 1.24.3-dev).
  • tools/estimateCellProportions.R (EpiDISH) with automatic M-value → Beta-value conversion for MESA-style inputs (1.23.1-dev).
  • Data-prep hardening: floor + log2(x+1) gene-expression transform in tecpg/mesa.py / tecpg/gtp.py, NaN diagnostics, early errors on <= -1 values, dataset-specific --logit-transform policy (1.23.0–1.23.2-dev).

5. Annotation Layer Rewrite

  • Comprehensive BED6 annotations for EPIC and HT-12 (hg19/hg38) (1.27.2-dev).
  • Rewritten tools/generate_annotations.py: multi-source HT-12 mapping with provenance (Re-Annotator → GEO → UCSC WG-6 → NA), canonical chromosome cleaning, retention of unmapped probes with NA coordinates, and a single write_bed6() gate. Net effect: HT-12 hg19 valid mappings rise 42,692 → 51,553. Includes 28 offline synthetic tests (1.27.1, 1.27.4-dev).
  • Vendored Re-Annotator HT-12 v4 data and UCSC hg19 illuminaProbes.txt recovery source (1.27.0-dev, post-1.27.6).
  • BED6 readers skip NA-coordinate rows safely (eliminates int('') and chr22.0 crashes) (1.27.5-dev).
  • Missing-annotation logging consolidated to a sidecar annotation_missing_ids.txt file instead of flooding the main log (1.27.6-dev).

6. Visualization Suite

  • Circos plot generator for eQTM architecture with cytoband labels, custom legend, and detailed exclusion reporting (missing/invalid coords, chromosome not in cytoband) (1.6.6–1.6.8, 1.23.0, 1.26.3–1.26.7-dev). Switched to hg19 cytoband (1.26.3-dev).
  • tools/visualizeFindings.py for multi-omic Volcano / Manhattan / Scatter plots, with per-p-value-column output prefixes (bootstrapP_, preciseP_, mtP_) preferring p_boot when available (1.6.3, 1.22.4, 1.23.0-dev).
  • Bipartite Network Suite: tools/exportBipartiteNetwork.py (formerly export_cytoscape.py) emits Cytoscape-style node/edge tables with effect-size / bootstrap-p / top-K filtering and --out-dir support (1.24.0, 1.24.4, 1.26.5-dev); tools/visualizeBipartiteNetwork.py adds ForceAtlas2 layouts, UMAP Beta-Diversity (Bray–Curtis), degree distributions, clustered adjacency heatmaps, arc diagrams, and bipartite→unipartite projection (count / sum / hypergeometric) (1.24.1–1.24.2-dev).
  • ENCODE enrichment analysis and regional functional enrichment (gseapy, mygene) in summarizeOutput_parquet.py; FDR threshold discovery, QQ plots, genomic-inflation (λ) (1.3.x, 1.6.8-dev).

7. Docker & Deployment

  • Multi-stage Docker rebuild against dev with full-pipeline support, runtime R + shared-library fixes (libxml2, libcurl4, libssl3), and a docker-related/REBUILD_DESIGN.md design memo (1.27.0, 1.27.3-dev).
  • tools/install_dependencies.R to install all required R packages (pheatmap, EpiDISH, sva, EPIC anno, ExperimentHub) via BiocManager (1.22.5-dev).

8. Profiling, Diagnostics & Agents

  • profiling.sh bash GPU diagnostic tool driving nvidia-smi / top / vmstat / pidstat, with matrix sweeps over prefetching, chunk size, TF32, and BLAS threads; environment-annotated result tarball (docs/profiling.md) (1.15.0–1.15.1-dev).
  • AGENTS.md and vendored Clean Code / A Philosophy of Software Design rule sets under docs/agent-rules/ for AI agent contributors (1.15.0-dev).
  • Gap-fill diagnostics in startup banner and per-chunk metrics (gpu_idle_between_chunks_ms, save_queue_depth, prefetch_fill) (1.13.0-dev).

Breaking Changes

  • CLI short flags removed. tecpg run mlr no longer accepts -g / -m for --gene-loci-per-chunk / --meth-loci-per-chunk — they collided with the top-level -g, --gene-file / -m, --meth-file. Use the long forms: --gene-loci-per-chunk <N> --meth-loci-per-chunk <N>. The data dummy and chunks subcommands' own -g/-m flags are unchanged. (1.21.0-dev)
  • Default p-thresh reverted to 0.001 (briefly 0.00001 in 1.22.3-dev) (1.24.2-dev).
  • tools/export_cytoscape.py renamed to tools/exportBipartiteNetwork.py (and its test file likewise) (1.24.1-dev).
  • Output format default is now auto (Parquet on server-class hosts, CSV on minimum-class); explicit --output-format overrides win (1.20.1-dev).

Notable Bug Fixes (highlights)

  • Severe OOM regression on IG runs with tight VRAM (1.22.2-dev).
  • pipeline.sh mlr_run.log clobber via tee inside the tecpg-managed output dir (1.22.3-dev).
  • Duplicate-edge ValueError in visualizeBipartiteNetwork.py UMAP/heatmap pivots; dropped rows persisted to dropped_duplicate_edges.csv (1.26.2-dev).
  • prefetch_chunks auto-resolution now reports 0 on CPU-only systems, preventing pin_memory() crashes (1.20.1-dev).
  • Bottleneck-log ETA correction (1.26.0-dev).
  • Numerous Parquet schema / index / nullable-int fixes (see §2).

New Dependencies

networkx, fa2, umap-learn, plus R packages installable via tools/install_dependencies.R.

Upgrade Notes

  • Update any scripts that pass -g <N> -m <N> to tecpg run mlr (use long forms).
  • If you used tools/export_cytoscape.py, switch to tools/exportBipartiteNetwork.py.
  • For minimum-spec hosts, set --host-profile minimum (or TECPG_HOST_PROFILE=minimum) to opt into conservative defaults; for tight-VRAM IG runs, the new safety ceiling is automatic.
  • Rebuild Docker images from the updated multi-stage Dockerfile to pick up runtime R/library fixes.

Full per-version detail is in CHANGELOG.md (versions 1.1.0-dev through 2.0.0b1).

Assets 2
Loading