Skip to content

Running the Pipeline

Ali Altintas edited this page Aug 26, 2026 · 1 revision

Running the Pipeline

Test profile

config/test.yaml runs the whole workflow end-to-end against a bundled synthetic dataset in .tests/ (50kb genome, 100 genes, one TE, 2 conditions x 3 replicates): lane-split paired-end samples exercise the lane-merging step, single-end samples exercise that branch too, and a DESeq2 contrast mixes both library formats. Useful for confirming your setup or smoke-testing rule edits:

snakemake --configfile config/test.yaml --cores 4 -n   # dry-run
snakemake --configfile config/test.yaml --cores 4      # run

Useful partial targets

snakemake --cores 8 star_index_only      # just build the STAR index
snakemake --cores 8 strandedness_only    # align + auto-detect strandedness only
snakemake --cores 8 trimming_only        # merge lanes + TrimGalore! only (no alignment)

Resuming & troubleshooting

  • Re-run after a failure: just run the same command again — Snakemake skips jobs it already finished. Add --rerun-incomplete if the failed job may have left a partial output file behind.
  • Force a specific step: snakemake -R <rule> re-runs a rule and its downstream (e.g. -R star_index to rebuild the index).
  • Where to look first: per-rule stdout/stderr lives in results/pipeline_info/logs/<rule>/...; scheduler-level errors in .snakemake/log/. For a failed SLURM job, sacct -j <jobid> shows the node and exit code.
  • "output … missing locally, parent dir not present": the job wrote to a node-local path (typically /tmp) the scheduler can't see from the submission node. Keep ref.decompressed_dir on shared storage (see Configuration Reference); a node-local or empty value is rejected at startup while gzipped references are in use.
  • An input change isn't picked up: rules with ancient() inputs (the STAR index) are only rebuilt when missing or via -R. The workflow warns at startup if the index was built for a different reference setup than the current config.
  • Multiple --configfile: Snakemake keeps only the last one — they are not merged. Merge settings by editing input/config.yaml, or pass single overrides with --config key=value.

Clone this wiki locally