Skip to content

Tutorial for ProcessND.py

jdkio edited this page Jul 2, 2026 · 23 revisions

ProcessND Tutorial (Updated for AL9)

⚠️ Warnings – Please Read First

βœ… Use NERSC if Possible

ProcessND.py is now outdated. The official way to generate samples is through NERSC. Contact #nd_production or #nd_reco_sim on Slack for the most up-to-date methods.

πŸ” Reuse Existing Samples

Before generating anything new, check if a suitable sample already exists. Coordination leads to better use of resourcesβ€”talk to others about shared sample needs.

⚠️ Get Approval for Large Jobs

Large jobs cost time and computing resources. Always get permission and notify #nd_production and/or #nd_reco_sim before launching grid jobs.

βœ… Always Run a Small Test First

Validate everything before scaling up. Some errors only show up at later stages (e.g., edep files exist even when genie failed). Review log files carefully.

βœ… AL9 Compatible (SL7 No Longer Required)

These scripts now run natively in AL9! SL7 containers are no longer required to run the production script. Still needed to compile dune-tms to run on the grid though.

πŸ“‚ Make Your Output Useful to Others

Save your samples in a persistent, accessible location. Document settings clearly for reproducibility.

❓ Why Does This Tutorial Still Exist?

  1. Sometimes it can take a while to get a NERSC sample
  2. Some features like pileup are still easier through grid jobs.

πŸ”§ Setup Instructions

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
cd <your local ND_Production clone>
cd toolbox/scripts

☎️ Download instructions

source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
cd /exp/dune/app/users/$USER/<your production subdir>
git clone git@github.com:DUNE/ND_Production.git
cd ND_Production

πŸ› οΈ Description of ProcessND.py

ProcessND.py is a wrapper script that creates processnd.sh, which you submit to the grid with jobsub_submit.


🧰 jobsub_submit Key Options

  • -N: Number of output files. If generating, total POT = POT per file Γ— N. If processing existing files, set -N to the number of input files.

  • --memory=4000MB: A good starting point. Double it for overlay.

  • --expected-lifetime=24h: Set as close to actual runtime as possible to improve scheduling.

  • --tar_file_name: Path to your dune-tms tarball. Tag and push your version in Git for reproducibility:

dune-tms should still be compiled in SL7 for compatibility with current worker nodes. Use:

/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
  -B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf \
  --ipc --pid \
  /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest

# then setup with setup.sh (sl7 setup), and compile
# Tag for reproducability
git tag <tag_name>
git push origin <tag_name>
# then go down one directory with cd .. and do the following
tar -czvf dune-tms.tar.gz dune-tms
mv dune-tms.tar.gz /pnfs/dune/persistent/users/$USER/dune-tms_tarfiles/your_tarfile.tar.gz


πŸ§ͺ Example: TMS-only Sample

python ProcessND.py --stages tmsreco \
  --indir /pnfs/dune/persistent/users/abooth/.../EDEPSIM_SPILLS/00000/ \
  --outdir /pnfs/dune/scratch/users/$USER/nd_production/my_test

jobsub_submit --group dune --role=Analysis -N 100 \
  --singularity-image /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl7:latest \
  --expected-lifetime=24h \
  --append_condor_requirements='(TARGET.HAS_Singularity==true&&TARGET.HAS_CVMFS_dune_opensciencegrid_org==true&&TARGET.HAS_CVMFS_larsoft_opensciencegrid_org==true&&TARGET.CVMFS_dune_opensciencegrid_org_REVISION>=1105)' \
  --memory=4000MB \
  --tar_file_name dropbox:///pnfs/dune/persistent/users/kleykamp/dune-tms_tarfiles/2025-02-13_Kalman_simplest_B_field.tar.gz \
  file://processnd.sh

πŸ§ͺ Example: Full Chain (gen + g4 + tmsreco)

python ProcessND.py \
  --outdir /pnfs/dune/scratch/users/$USER/nd_production/full_chain \
  --geometry_location /pnfs/dune/persistent/physicsgroups/dunendsim/geometries/TDR_Production_geometry_v_1.0.3/nd_hall_with_lar_tms_sand_TDR_Production_geometry_v_1.0.3.gdml \
  --manual_geometry_override nd_hall_with_lar_tms_sand_TDR_Production_geometry_v_1.0.3.gdml \
  --topvol volDetEnclosure \
  --pot 1e15 \
  --stages gen+g4+tmsreco
  • Use volDetEnclosure to limit to LAr+TMS+Sand. Avoid volWorld unless simulating full rock detector.

πŸƒ Running Locally

One can run locally with the --test flag. This puts everything in the test directory locally but will otherwise be the same. Make sure to run inside an SL7 container as if you're the grid:

# Start SL7 container
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/current/bin/apptainer shell --shell=/bin/bash \
  -B /cvmfs,/exp,/nashome,/pnfs/dune,/opt,/run/user,/etc/hostname,/etc/hosts,/etc/krb5.conf \
  --ipc --pid \
  /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest
# Run the command created with ProcessND.py, make sure to use the --test flag
./processnd.sh

πŸ”„ Optional Modifications

πŸ” Run Only Genie + EDepSim

--stages gen+g4

🧊 LAr-Only Simulation

--topvol volLArBath

Refer to the [geometry viewer](https://dune.github.io/dunendggd/) and Slack channel #nd_geometry for help.

🌊 Overlay (Pileup)

--timing spill --overlay
  • Increase memory to 8000MB at least.

↔️ Off-Axis Simulation

--oa <distance> --dk2nu
  • gsimple flux files do not support off-axis info.

🧲 Magnetic Field Variants

B-field support is possible but not fully validated in this script. Ask around if you need this.


⚠️ Known Gotchas

❌ Wrong Working Directory

Run from inside the scripts/ directory:

cd scripts
python ProcessND.py

Otherwise it won’t find template.sh.


Running a particle gun

See instructions on slack and this tutorial.


Questions? Ask in #nd_production, #nd_muon_spectrometer, or #nd_muon_spectrometer_code on Slack.