tests_benchmark — CPU vs GPU optical-photon benchmark

Drives edep-sim (CPU optical transport) and the eic-opticks GPU plugin on one shared nested-shell LArTPC geometry, compares their scintillation light, and provides a 3D event display.

• Geometry: 60 × 13.5 × 13 m active LAr (14.7 kt), wrapped in 3 concentric shells — 200 µm pTP → 6 mm TPB acrylic → 1 mm idealized 100 % photon-detector shell. Built by the local benchmark_ggd/ gegede package (see below).
• Primary (default): a single 1 MeV e⁻ at the detector centre, fired +x.
• The plugin lives in $OPTIC_GPU_ROOT/edep-simphony-plugin/ (its own git repo). All GPU behaviour is controlled by EDEP_SIMPHONY_* environment variables, documented below.

benchmark_ggd/     gegede geometry package (own git repo) -> benchmark.gdml
run.sh             the single driver (modes: both/cpu/gpu/dual/photon/photon1/dualtraj/analyze)
run_*.mac          one Geant4 macro per mode
*.py               analysis / comparison / plotting scripts
viz/               self-contained 3D event display (see viz/README.md)
output/            all ROOT files, logs, plots, and the viewer JSON

---

0. One-time setup

First set the machine-specific paths. run.sh reads them from config.sh, which is git-ignored — copy the template and point it at your checkout:

cp config.example.sh config.sh
# edit config.sh:  export OPTIC_GPU_ROOT=/your/path/to/optic-gpu

config.sh is the only place absolute paths live; everything else (PLUGIN_ENV, BASE_ENV, the GDML, the G4Scintillation override) is derived from OPTIC_GPU_ROOT. You can also just export OPTIC_GPU_ROOT=... in your shell instead of creating config.sh.

Then load the base environment:

source $OPTIC_GPU_ROOT/.envrc          # base env (ROOT, gegede, edep-sim)
# the plugin env is sourced automatically by run.sh for GPU modes

Build the geometry GDML (only needed if it's missing or you edit the geometry):

cd tests_benchmark/benchmark_ggd
python build_geo.py            # -> benchmark.gdml  (run.sh expects it here)

run.sh finds the GDML at tests_benchmark/benchmark_ggd/benchmark.gdml automatically.

---

1. Running — the modes

All runs go through run.sh <N> <mode> (N = number of events). Output lands in output/<mode>.root + output/<mode>.log.

cd tests_benchmark

./run.sh 100 both       # CPU run + GPU run separately, then comparison.png
./run.sh 100 cpu        # CPU only            -> cpu.root
./run.sh 100 gpu        # GPU only            -> gpu.root
./run.sh 100 dual       # CPU+GPU in ONE run  -> dual.root  (per-event compare)
./run.sh 100 photon     # 100 pure 128nm photons/event, CPU+GPU -> photon.root
./run.sh 1000 photon1   # 1 pure photon/event, CPU+GPU          -> photon1.root
./run.sh 1   dualtraj   # 1 MeV e-, CPU+GPU, FULL trajectories  -> dual_traj.root
./run.sh 100 analyze    # just re-run analyze.py on existing ROOT files

Charged-particle vs photon-only runs

• Charged particle (scintillation light) — cpu, gpu, dual, dualtraj. These fire the primary from run_*.mac (default 1 MeV e⁻). The electron ionises the LAr, scintillation photons are generated, and CPU/GPU transport them. To change the particle, edit the /gps/... block in the macro, e.g. in run_dual_traj.mac:

  /gps/particle mu-        # muon instead of electron
  /gps/energy 500 MeV      # longer track -> richer charge + far more light
  /gps/position 0 0 0 mm
  /gps/direction 1 0 0
  

• Photon-only — photon (100 photons/event) and photon1 (1 photon/event). These fire pure optical photons directly (no charged particle, no scintillation), at the LAr scintillation wavelength (128 nm = 9.6863 eV), and transport the identical photon population on CPU and GPU for a clean transport comparison. Photon kinematics live in run_photon.mac / run_photon1.mac (/gps/particle opticalphoton, /gps/number, etc.).

---

2. GPU controls (EDEP_SIMPHONY_* environment variables)

These are read by the plugin at runtime. run.sh sets sensible defaults per mode, but any of them can be overridden on the command line: EDEP_SIMPHONY_MAXBOUNCE=500 ./run.sh 1 dualtraj.

Variable	Meaning	Default in run.sh
EDEP_SIMPHONY_MAXBOUNCE	Max GPU bounces per photon (see below)	1000 (dual/traj/photon1)
EDEP_SIMPHONY_MAXRECORD	Length of the saved per-photon GPU trajectory; must be ≥ MaxBounce+1 to store full paths	MaxBounce+1
EDEP_SIMPHONY_MAXSLOT	GPU photon-slot allocation (record buffer = MaxSlot × MaxRecord)	20000 (dualtraj); env OPTICKS_MAX_SLOT otherwise
EDEP_SIMPHONY_DUAL	1 = CPU and GPU transport in one run (see §3)	unset (separate runs)
EDEP_SIMPHONY_DEBUGHEAVY	1 = GPU records every photon + full bounce-by-bounce trajectory (see §4)	unset; 1 in dualtraj
EDEP_SIMPHONY_INPUT_PHOTONS	1 = inject the CPU primary optical photons into Opticks as input photons (photon-only modes)	set in photon/photon1
EDEP_SIMPHONY_CERENKOV	0 = disable Cerenkov (scintillation-only comparison)	0 in all comparison modes
EDEP_SIMPHONY_SCINT	scint process: thin (SimphonyScintProcess, default) or fork (Local_DsG4Scintillation)	thin

Setting the GPU max-bounce cap

128 nm photons Rayleigh-scatter heavily in LAr, so they need a high bounce budget. The cap is the single most impactful GPU knob: too low and photons hit the limit ("Scatter cap") before being absorbed/detected, artificially lowering GPU efficiency.

# raise / lower the GPU bounce cap (run.sh sets MaxRecord = MaxBounce+1 for you)
EDEP_SIMPHONY_MAXBOUNCE=1000 ./run.sh 1 dualtraj      # generous (default)
EDEP_SIMPHONY_MAXBOUNCE=32   ./run.sh 1 dualtraj      # tight cap, for cap studies

NOTE: the GPU record array is also bounded by Opticks sseq::SLOTS (currently 112, baked into simphony/sysrap/sseq.h). To record paths longer than ~111 steps you must raise BOTH EDEP_SIMPHONY_MAXBOUNCE (env) AND sseq::SLOTS (recompile simphony). The CPU side has no bounce cap and always propagates until physically absorbed/detected.

---

3. DUAL mode — saving CPU and GPU light in one file

Enable: pass dual (or dualtraj) as the mode, OR set EDEP_SIMPHONY_DUAL=1 in any GPU run. In dual mode the plugin:

• adds a stock G4Scintillation (stacking on) so the CPU tracks photons to the SurfaceSD → PhotonDetectors, AND
• runs the GPU genstep path → GPUPhotonHits,

both written to the same dual.root for per-event CPU-vs-GPU comparison.

./run.sh 100 dual                  # EDEP_SIMPHONY_DUAL=1 is set internally
python3 dual_per_event.py          # per-event nphoton/nhit table
python3 plot_dual_hits.py          # aggregate hit comparison plot

Disable: just use cpu or gpu mode (separate runs, no dual scintillation), or leave EDEP_SIMPHONY_DUAL unset — the default plugin behaviour is GPU-only with CPU photons killed.

---

4. Trajectory saving (CPU and GPU) — enable / disable

There are two independent trajectory recorders; both must be on to get full paths on both sides.

CPU trajectories (edep-sim)

Controlled by macro commands (NOT env vars), placed BEFORE /edep/update:

/edep/db/set/savePhotonTraj  true     # save optical-photon trajectories
/edep/db/set/saveAllPrimTraj true     # save all primary-related trajectories
/edep/db/set/saveAllTraj 0 MeV        # save EVERY charged trajectory incl. delta rays

• Enable: set the above to true (and saveAllTraj 0 MeV for charge). This is the case in run_dual_traj.mac and run_photon*.mac.
• Disable: set savePhotonTraj false + saveAllPrimTraj false (as in run_dual.mac) — used for high-stats runs where trajectories would bloat the file. (There is no /edep/db/set/saveTrajectory command.)

The plugin's CPU recorder user-actions additionally fill CPUPhotonTracks (per-photon fate) and CPUPhotonSteps (full per-photon path), loaded via the /edep/actions/loadUser* lines in the macro.

GPU trajectories (eic-opticks)

Controlled by env vars:

EDEP_SIMPHONY_DEBUGHEAVY=1                 # record ALL photons + full bounce path
EDEP_SIMPHONY_MAXRECORD=$((MaxBounce+1))   # buffer long enough to hold them
EDEP_SIMPHONY_MAXSLOT=20000                # enough photon slots (watch VRAM!)

• Enable: EDEP_SIMPHONY_DEBUGHEAVY=1 (or INPUT_PHOTONS=1) → fills GPUPhotonTracks (per-photon fate) and GPUPhotonSteps (full path). This is set automatically in dualtraj.
• Disable: leave EDEP_SIMPHONY_DEBUGHEAVY unset → GPU records only GPUPhotonHits (detected), no full trajectories. Use for high-stats runs; DebugHeavy is memory-hungry (record alloc = MaxSlot × MaxRecord) and is only safe for few-photon runs.

The dualtraj mode turns ON all four: CPU savePhotonTraj/saveAllTraj + GPU DebugHeavy + DUAL. That is the only mode that produces complete CPU AND GPU trajectories for the same event.

---

5. Making a trajectory file viewable in the 3D viewer

The viewer reads a JSON extracted from a trajectory ROOT file. After a dualtraj run:

# 1. produce a trajectory file (full CPU + GPU paths + charge)
./run.sh 1 dualtraj                        # -> output/dual_traj.root

# 2. extract the particle flow -> viewer JSON (+ slimmed ROOT files)
source $OPTIC_GPU_ROOT/.envrc
python3 viz/extract_event.py               # reads output/dual_traj.root
#   writes output/event_view.json(.gz)   <- the ONLY thing the viewer reads
#          output/light_cpu.root         <- slimmed: geometry+charge+CPU light
#          output/light_gpu.root         <- slimmed: geometry+charge+GPU light

# 3. open the viewer
bash viz/serve.sh                          # http://localhost:8000/viewer.html

To extract from a different trajectory file: python3 viz/extract_event.py path/to/your.root (it must contain the EDepSimEvents, CPUPhotonTracks/Steps, GPUPhotonTracks/Steps, GPUPhotonHits, and EDepSimGeometry objects — i.e. produced by a dualtraj run with all trajectory recorders on, §4).

The viewer has no ROOT dependency — it loads only event_view.json (or .json.gz). The two light_*.root files are separate, canonical outputs for downstream ROOT/edep-sim tools, not used by the viewer. See viz/README.md for the controls (light %, fixed-count, geometry planes, XY/XZ/YZ projections, double-click-to-zoom, etc.).

A high-stats dual.root (trajectories OFF) is hits-only and is NOT directly viewable — use a dualtraj run for the display.

---

6. Analysis / plotting scripts

(Source the base env first.)

python3 analyze.py                                   # both -> comparison.png + stats
python3 compare_photon.py output/photon.root out     # photon-mode CPU/GPU fate compare
python3 compare_photon.py output/dual_traj.root output/dual_traj_compare "Scint from e-"
python3 dual_per_event.py                            # per-event CPU/GPU table (dual.root)
python3 plot_dual_hits.py                            # aggregate dual hit comparison
python3 plot_cap_compare.py                          # MaxBounce 32 vs 1000 cap study
python3 plot_numphotons.py                           # per-event generated-photon plot

---

7. Geometry package (benchmark_ggd/)

A self-contained gegede package (its own git repo, remote Ningclover/benchmark_ggd) that builds benchmark.gdml:

cd benchmark_ggd
python build_geo.py            # -> benchmark.gdml

Key GDML auxiliaries (set in benchmark.py):

• SurfaceDetector "PhotonDetector" on the outer shell (REFLECTIVITY=0, EFFICIENCY=1 → idealized 100 % photon counter).
• SegmentDetector "lar" on the active LAr → records charge (ionisation) dE/dx as TG4HitSegments (the charge half of the event).
• EField (500 V/cm,0,0) on the LAr → DokeBirks recombination per step.
• four *CONSTANT matrices + state="liquid" on G4_lAr (GPU scintillation yield; the state is patched in post by build_geo.py).

After editing the geometry, re-run python build_geo.py, then re-run the sim.

---

Rebuilding the plugin (after editing its src/)

source $OPTIC_GPU_ROOT/edep-simphony-plugin/setup_env.sh
cd $OPTIC_GPU_ROOT/edep-simphony-plugin/build && make -j4

Rebuilding simphony itself (e.g. after changing sseq::SLOTS):

cd $OPTIC_GPU_ROOT/simphony && source ../.envrc
cmake --build build -j8 && cmake --install build