Troubleshooting

Troubleshooting & Notes

Cross-validation

• 12-fold leave-chromosome-out. Always report the pooled out-of-fold Pearson R — concatenate the held-out predictions from all 12 folds into one vector and compute a single R. Do not average per-fold R (it differs and is not comparable).

Output directories and evaluation protocols

• download_encode_data.py defaults to downloads/; pass --output-dir data because config/samples.tsv expects data/{cell}/....
• Keep RNA, CAGE, reverse-only, fwd+RC, and model variants in separate prediction directories. evaluate.py rejects multiple files for one fold to prevent accidental pooling.
• For the pinned K562 narrowPeak, single-reverse evaluation is approximately 0.734; the headline approximately 0.740 value uses forward-plus-reverse-complement averaging.

HPC safety

• Run run_pipeline.py --dry-run on a login node, then submit actual links/encoding work with SAMPLES=... STAGES=... sbatch slurm/run_pipeline_cpu.slurm.
• Full encoder and expression CV use the 12-task GPU array scripts. The notebook training cells are for inspecting one fold, not a substitute for the full arrays.

Enhancer-activity target

• The encoder target is log2(0.1 + Activity) — base-2 log, pseudocount 0.1, not log1p. Because of the pseudocount, ~77% of K562 windows are negative (log2 = 0 ↔ Activity ≈ 0.9 RPM), so a barplot of raw predictions being mostly ≤ 0 is expected for background sequence.
• The promoter-signal feature (--use_prm_signal) uses natural log(1 + x) — a different transform.

Empty-promoter NaN at training time

Genes whose promoter sequence is all-N (near assembly gaps) fully mask the attention, giving a softmax over -inf → NaN at epoch-0 validation. The fix (already in EPInformer_v2.forward) forces the promoter slot to always be attendable:

enhancers_padding_mask[:, 0] = False   # slot 0 is the promoter — never fully masked

This is a no-op for K562/GM12878 (0 empty promoters) and required for H1/HepG2/HUVEC/NHEK.

hg19 vs hg38 coordinates

The repo's genome and enhancer lists are hg38. Upstream's predict_enhancer_activity.ipynb KLF1 coordinates are hg19 (from Fulco-2019 CRISPRi-FlowFISH). Feeding an hg19 coordinate to the hg38 FASTA lands ~90 kb off the target on background sequence (all-negative predictions). If you scan a locus, make sure the coordinate and the FASTA are the same build. github_enhancer_activity_demo.ipynb uses hg19 throughout; predict_enhancer_and_expression.ipynb uses hg38.

ISM (tangermeme)

In tangermeme ≥ 1.x the module is tangermeme.saturation_mutagenesis (older code imported tangermeme.ism). The saturation_mutagenesis function is otherwise the same and returns a (1, 4, L) tensor, so seq_ISM[0] feeds tangermeme.plot.plot_logo directly.

Executing the notebooks headlessly

jupyter nbconvert --to notebook --execute needs a kernel that has torch / kipoiseq (register one with python -m ipykernel install --user --name <env> and pass --ExecutePreprocessor.kernel_name=<env>). Use the inline matplotlib backend — do not set MPLBACKEND=Agg, or plt.show() embeds nothing.