Skip to content

ECG Slurm Runs

Jump to bottom
Abdullah edited this page May 26, 2026 · 2 revisions

ECG Slurm Final Runs

This page describes the reproducible UVA Slurm workflow for ECG final-paper runs. The goal is to split slow gem5 or Sniper simulations into independent shards, run each shard on a separate machine, and aggregate all CSVs later.

The key rule: one GraphBrew gem5 process per node unless you have explicitly isolated the runtime sideband files. The gem5 path uses shared sideband files under /tmp, so Slurm jobs should request exclusive nodes or otherwise ensure only one GraphBrew gem5 shard runs per node.

For autonomous execution, follow the ECG autonomous final-paper completion task. It keeps build checks, dry-runs, tiny smokes, and shard generation local, while large file-backed gem5/Sniper/final profiles are saved as Slurm shards instead of being launched on the workstation.

Overview

The workflow has four phases:

  1. Prepare a shared checkout, Python environment, gem5 build, benchmark binaries, and graph data.
  2. Generate one shard per graph / benchmark / policy.
  3. Submit Slurm array jobs, where each task runs exactly one shard.
  4. Aggregate completed shard directories with paper_pipeline.py --skip-run.

The final-run wrapper supports these sharding filters:

--graph <name>          # exact normalized manifest graph name
--benchmark <name>      # pr, bfs, or sssp
--policy <label>        # LRU, POPT_CHARGED, ECG_DBG_PRIMARY, ...
--only <stage-token>    # optional stage filter

For example, this lists exactly one replacement shard:

python3 scripts/experiments/ecg/final_paper_run.py \
  --profile final_replacement \
  --run-dir results/ecg_experiments/final_paper_runs/slurm_dryrun \
  --graph soc-pokec \
  --benchmark pr \
  --policy ECG_DBG_PRIMARY_CHARGED \
  --list \
  --dry-run \
  --allow-missing-graphs \
  --skip-validation-gate

1. Prepare the UVA Slurm Workspace

Use shared storage visible to all compute nodes, such as $SCRATCH or a project allocation. Replace module names with the currently available UVA modules.

# Example layout. Adjust to your allocation.
export GRAPHBREW_ROOT=$SCRATCH/GraphBrew
export GRAPHBREW_RUNS=$SCRATCH/graphbrew_ecg_runs
export GRAPHBREW_GRAPHS=$GRAPHBREW_ROOT/results/graphs

mkdir -p "$GRAPHBREW_RUNS" "$GRAPHBREW_GRAPHS" results/slurm_logs
cd "$GRAPHBREW_ROOT"

module purge
module avail gcc python
module load gcc/<version> python/<version>

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r scripts/requirements.txt
pip install 'scons==4.6.0'

Build gem5 once, not inside every Slurm task:

source .venv/bin/activate
python3 scripts/setup_gem5.py --isa X86
make -j8 all-sim
make gem5-m5ops-pr gem5-m5ops-bfs gem5-m5ops-sssp

If gem5 has already been built on shared storage, do not rebuild it in worker jobs. The final-run manifest defaults to --no-build.

2. Stage Graph Data

Final profiles expect these paths:

results/graphs/soc-pokec/soc-pokec.sg
results/graphs/soc-LiveJournal1/soc-LiveJournal1.sg
results/graphs/com-orkut/com-orkut.sg
results/graphs/cit-Patents/cit-Patents.sg

For available-local smoke runs, these paths are used:

results/graphs/email-Eu-core/email-Eu-core.sg
results/graphs/cit-Patents/cit-Patents.sg

Preferred graph-staging pattern:

mkdir -p results/graphs/<graph-name>
# Put raw graph as .el, .mtx, or .txt after removing source-specific headers.
# Then convert to serialized GAP/GraphBrew format.
make converter
bench/bin/converter \
  -f results/graphs/<graph-name>/<raw-file>.el \
  -s \
  -b results/graphs/<graph-name>/<graph-name>.sg

For compressed edge-list downloads:

mkdir -p results/graphs/cit-Patents
zcat /path/to/cit-Patents.txt.gz | grep -v '^#' > results/graphs/cit-Patents/cit-Patents.el
bench/bin/converter \
  -f results/graphs/cit-Patents/cit-Patents.el \
  -s \
  -b results/graphs/cit-Patents/cit-Patents.sg

For Matrix Market downloads:

mkdir -p results/graphs/<graph-name>
bench/bin/converter \
  -f /path/to/<graph-name>.mtx \
  -s \
  -b results/graphs/<graph-name>/<graph-name>.sg

After staging, verify exactly what the manifest will see:

python3 scripts/experiments/ecg/final_paper_run.py \
  --profile final_replacement \
  --check-graphs \
  --allow-missing-graphs

python3 scripts/experiments/ecg/final_paper_run.py \
  --profile final_droplet \
  --check-graphs \
  --allow-missing-graphs

Do not start the multi-day run until missing graphs are resolved or intentionally excluded with --graph filters.

3. Generate Slurm Shard List

Create one tab-separated row per shard from the checked-in final-run manifest. The generated TSV is headerless because Slurm array task 0 reads row 1.

mkdir -p results/ecg_experiments/slurm
RUN_TAG=final_$(date +%Y%m%d_%H%M%S)
SHARDS=results/ecg_experiments/slurm/${RUN_TAG}_shards.tsv

python3 scripts/experiments/ecg/make_slurm_shards.py \
  --profile final_replacement final_droplet \
  --run-tag "$RUN_TAG" \
  --out "$SHARDS"

wc -l "$SHARDS"
sed -n '1,5p' "$SHARDS"

The full four-graph PR/BFS/SSSP replacement+DROPLET set above creates:

4 graphs * 3 benchmarks * (9 replacement policies + 7 DROPLET policies) = 192 Slurm shards

For a smaller first pass, use the generator filters. For example:

python3 scripts/experiments/ecg/make_slurm_shards.py \
  --profile final_replacement \
  --run-tag "$RUN_TAG" \
  --graph cit-Patents \
  --benchmark pr \
  --policy LRU ECG_DBG_PRIMARY \
  --out "$SHARDS"

For the current long Sniper scale-out gate, generate a one-row Slurm shard with:

python3 scripts/experiments/ecg/make_slurm_shards.py \
  --profile sniper_sift_cit_patents_long \
  --run-tag "$RUN_TAG" \
  --out "$SHARDS"

4. Slurm Array Script

The repository includes scripts/experiments/ecg/slurm_final_shard.sbatch. Replace account, partition, time, and memory with the UVA allocation you are using.

Review it before submitting:

sed -n '1,120p' scripts/experiments/ecg/slurm_final_shard.sbatch

Submit the array:

RUN_TAG=final_20260524_000000
SHARDS=results/ecg_experiments/slurm/${RUN_TAG}_shards.tsv
N=$(( $(wc -l < "$SHARDS") - 1 ))

export GRAPHBREW_ROOT=$SCRATCH/GraphBrew
export SHARDS
sbatch --array=0-${N} scripts/experiments/ecg/slurm_final_shard.sbatch

Use --exclusive or an equivalent UVA policy so Slurm does not place two GraphBrew gem5 shards on the same node. If your allocation does not allow exclusive nodes, limit the array to one task per node by partition policy or run cache_sim shards separately from gem5 shards.

5. Monitor and Resume

Check Slurm status:

squeue -u "$USER" -n ecg-gem5
sacct -j <jobid> --format=JobID,JobName,State,Elapsed,MaxRSS,ExitCode

Check a shard run directory:

python3 scripts/experiments/ecg/final_paper_run.py \
  --status \
  --run-dir results/ecg_experiments/final_paper_runs/slurm/${RUN_TAG}/final_replacement_soc-pokec_pr_LRU

Summarize every row from the shard TSV without launching simulations:

python3 scripts/experiments/ecg/slurm_shard_status.py \
  --shards "$SHARDS" \
  --out results/ecg_experiments/slurm/${RUN_TAG}_status.csv

column -s, -t < results/ecg_experiments/slurm/${RUN_TAG}_status.csv | sed -n '1,20p'

Use --fail-on-failed in CI/check scripts when any failed shard should make the status command return nonzero. Use --fail-on-missing when pending or not-started rows should also fail the check.

Rerun a failed shard by resubmitting the same row. The wrapper is resumable: existing ok CSV rows are skipped unless --force is passed.

6. Aggregate Shards Later

After Slurm jobs finish, aggregate all shard directories without launching simulations:

python3 scripts/experiments/ecg/paper_pipeline.py \
  --skip-run \
  --input-run-glob "results/ecg_experiments/final_paper_runs/slurm/${RUN_TAG}/*" \
  --run-root "results/ecg_experiments/paper_pipeline/${RUN_TAG}_aggregate"

The aggregate run writes:

aggregate/roi_matrix_all.csv
aggregate/roi_relative_metrics.csv
aggregate/faithfulness_summary.csv
aggregate/popt_storage_overhead_summary.csv
aggregate/prefetch_quality_summary.csv
figures/*.svg
figures/*.png
tables/*.tex

If some shards are still running or failed, aggregation still works on completed shard directories. The missing policies/graphs simply will not appear in the combined CSVs.

7. Recommended UVA Run Order

  1. Run rehearsal locally or in a short Slurm job.
  2. Run available_replacement on the two locally available graph files.
  3. Stage all final graph .sg files.
  4. Run final_cache_sim shards if you want fast policy-direction checks.
  5. Run final_replacement shards.
  6. Run final_droplet shards after replacement shards are stable.
  7. Aggregate with paper_pipeline.py --skip-run --input-run-glob ....

Guardrails

  • Do not run multiple GraphBrew gem5 shards on the same node unless sideband files are isolated.
  • Keep graph data and run outputs on shared storage visible to all nodes.
  • Build gem5 and benchmark binaries once before submitting the array.
  • Use exact manifest graph names: soc-pokec, soc-LiveJournal1, com-orkut, cit-Patents.
  • Use exact normalized policy labels for --policy: LRU, SRRIP, GRASP, POPT_CHARGED, POPT, ECG_DBG_ONLY, ECG_DBG_PRIMARY_CHARGED, ECG_DBG_PRIMARY, ECG_POPT_PRIMARY.
  • POPT is the oracle ceiling; POPT_CHARGED is the honest prior-method baseline.

Clone this wiki locally