step-by-sample

A Bash-first pattern for per-sample workflow steps where you always:

1. generate one job script per sample,
2. collect them in a run list,
3. execute that run list locally or on Slurm.

Each sample writes its own run.log and ends with either .done or .failed in its output folder. The design stays intentionally small: no workflow engine, no background controller, just generated scripts plus helper commands.

Core Model

Every workflow step follows the same structure:

• one input directory,
• one output directory,
• one subdirectory per sample,
• one generated job script per sample,
• one run list file containing those job script paths.

This keeps reruns, debugging, and local-vs-Slurm execution consistent.

Quickstart

1. Copy the template:

cp examples/build-jobs-template.sh build-my-step.sh

1. Edit the template:

• set IN, OUT, JOB_DIR, and LIST,
• define how to find the sample input files,
• replace the example command with the real tool invocation.

In practice, you will usually rename all four path settings together so they match your real step. For example, if the step is trimming reads, use names like trimmed-output, jobs-trim-reads, and run-trim-reads.txt instead of the generic placeholders.

1. Generate jobs:

./build-my-step.sh

Each run rewrites run-my-step.txt from scratch based on the current MODE and FORCE settings. It does not append to an older list.

1. Execute the run list locally:

helpers/run-list-local.sh run-my-step.txt 4

Here, 4 means run up to four sample jobs in parallel on the local machine. You pass the run-list file, not JOB_DIR, because run-my-step.txt already contains the paths to the generated job scripts.

1. Or execute the same run list on Slurm:

helpers/run-list-slurm.sh run-my-step.txt \
  --account my_account \
  --partition cpu \
  --time 08:00:00 \
  --mem 16G \
  --cpus 8 \
  --array-max 20

1. Check results:

helpers/summarize-status.sh my-step-output

Folder Convention

At a given step, input and output usually look like this:

IN/
  sample1/
    ...
  sample2/
    ...

OUT/
  sample1/
    run.log
    .done or .failed
    ...
  sample2/
    run.log
    .done or .failed
    ...

Workflow Template

The only workflow template is examples/build-jobs-template.sh.

It generates:

• a job directory such as jobs-my-step/,
• a run list such as run-my-step.txt,
• one executable script per sample.

The run list contains absolute paths to those generated job scripts, which is why the local and Slurm helpers only need the run-list file.

That gives you:

• inspection before execution,
• easy single-sample reruns,
• the same generated work unit for local and Slurm execution,
• selective rebuilds with MODE=unfinished, MODE=failed, or MODE=all,
• no hidden state beyond filesystem markers and logs.

Selection rule: MODE controls which samples are normally included, while FORCE=1 overrides that filtering and rebuilds jobs for all samples.

Template Workflow

Generate job scripts:

./build-my-step.sh

Inspect what was built:

cat run-my-step.txt
ls jobs-my-step/

Run one sample directly if needed:

bash jobs-my-step/sample2.sh

Execution Helpers

Local execution

helpers/run-list-local.sh executes a run list with configurable parallelism:

helpers/run-list-local.sh run-my-step.txt 4

In that command, 4 is the number of sample jobs to run at the same time. The helper reads script paths from run-my-step.txt, so you do not pass jobs-my-step/ separately.

Slurm execution

helpers/run-list-slurm.sh submits the same run list as a Slurm array job:

helpers/run-list-slurm.sh run-my-step.txt \
  --account my_account \
  --partition cpu \
  --time 08:00:00 \
  --mem 16G \
  --cpus 8 \
  --array-max 20

If your cluster needs environment setup before module load, use --setup-file and one or more --module options:

helpers/run-list-slurm.sh run-my-step.txt \
  --account my_account \
  --partition cpu \
  --setup-file /etc/profile.d/modules.sh \
  --module my-tool/1.2.3 \
  --module python/3.11

--setup-file is sourced inside each Slurm task, not just in the shell that submits the job. --array-max limits how many array tasks may run at once; it does not limit the total number of samples in the submission.

Utility Helpers

• helpers/validate-step.sh: validate input directory structure before running.
• helpers/summarize-status.sh: report done/failed/other counts.
• helpers/repair-failed.sh: remove .failed markers and optionally clean partial outputs.
• helpers/common.sh: shared shell utilities for templates and custom scripts.

Reruns and Recovery

Rerun one sample directly:

bash jobs-my-step/sample2.sh

Rebuild the run list for failed samples only:

MODE="failed" ./build-my-step.sh
helpers/run-list-local.sh run-my-step.txt 4

Use MODE="failed" when you only want samples with an existing .failed marker. Use MODE="unfinished" when you want anything not yet marked .done, which includes previously failed samples and samples that have never been run.

Rebuild for unfinished samples:

MODE="unfinished" ./build-my-step.sh
helpers/run-list-local.sh run-my-step.txt 4

Clean up failed outputs before rebuilding:

helpers/repair-failed.sh my-step-output --clean-outputs
MODE="unfinished" ./build-my-step.sh
helpers/run-list-local.sh run-my-step.txt 4

Force a full rebuild of all sample jobs:

FORCE=1 ./build-my-step.sh

Project Structure

step-by-sample/
├── README.md
├── .gitignore
├── examples/
│   └── build-jobs-template.sh      # Workflow template
├── helpers/
│   ├── common.sh                   # Shared utility functions
│   ├── run-list-local.sh           # Local execution runner
│   ├── run-list-slurm.sh           # Slurm array submitter
│   ├── summarize-status.sh         # Status reporter
│   ├── validate-step.sh            # Input validation
│   └── repair-failed.sh            # Failed sample cleanup
└── tests/
    ├── run-all-tests.sh            # Main test runner
    ├── test-01-workflow.sh         # Core workflow template tests
    ├── test-02-helpers.sh          # Helper utility tests
    ├── test-03-edge-cases.sh       # Edge case tests
    ├── test-04-reruns.sh           # Rerun and recovery tests
    ├── lib/                        # Test helpers and mocks
    └── mock-slurm/                 # Mock Slurm for local testing

Testing

Run the full suite:

cd tests
./run-all-tests.sh

Run with verbose output:

./run-all-tests.sh --verbose

Run the quick subset:

./run-all-tests.sh --quick

Run a specific file:

./run-all-tests.sh "test-01*"

Coverage includes:

• core workflow generation and execution,
• helper utilities,
• rerun and recovery behavior,
• edge cases such as spaces, symlinks, and large sample counts,
• local testing of Slurm submission behavior.

See tests/README.md for the full testing guide.

Best Practices

Before running:

1. Validate the input structure with helpers/validate-step.sh INPUT_DIR.
2. Generate the per-sample job scripts and inspect them before executing.
3. Test locally on a small subset before using Slurm.

During development:

1. Keep your edited template under version control.
2. Check OUTPUT_DIR/sample_name/run.log first when debugging.
3. Use helpers/summarize-status.sh frequently while iterating.

For production:

1. Set explicit Slurm resources instead of relying on defaults.
2. Use --setup-file and --module when your cluster environment requires it.
3. Keep rerun behavior explicit by rebuilding with MODE=failed or MODE=unfinished.

Notes

• Scripts use #!/usr/bin/env bash and set -euo pipefail.
• Generated run lists contain absolute job script paths so they can be executed from any working directory.
• The Slurm helper submits and exits; per-sample job scripts create the status markers.
• During job generation, missing inputs usually cause a sample to be skipped because no runnable job can be built. Inside a generated job, missing inputs are treated as a failure and produce .failed.
• Docker examples in the template assume mounting $(pwd) to /work.
• The template uses intentionally generic names like my-step-output, jobs-my-step, and run-my-step.txt; rename them to match your real step.