diff --git a/.claude/commands/parallel-audit.md b/.claude/commands/parallel-audit.md new file mode 100644 index 0000000..18ee877 --- /dev/null +++ b/.claude/commands/parallel-audit.md @@ -0,0 +1,148 @@ +--- +description: Spawn N parallel Explore agents on independent slices of the repo, then synthesize findings into a prioritized punch list +argument-hint: [--agents N] +allowed-tools: Agent +--- + +# Parallel codebase audit + +Fan out `N` Explore subagents on **independent** slices of the repo, +each producing a tight punch list, then synthesize the returns into a +single prioritized list with `[critical|high|medium|low]` tags and +file:line citations. + +This pattern is the right tool when: +- The question spans the codebase (>3 directories or surfaces). +- The agents' work is mostly independent — overlap wastes tokens. +- The user wants concrete, citable findings, not a tour. + +It is **not** the right tool when the answer is in one file +(use Read), one symbol (use Bash grep), or already known +(answer directly). + +## Usage + +``` +/parallel-audit [--agents N] +``` + +- `` — what the audit is about. Examples: + `"simplification opportunities"`, + `"unused code and dead imports"`, + `"test coverage gaps in utils/"`, + `"CLI ergonomic inconsistencies"`, + `"performance hot paths"`, + `"security review of subprocess calls"`. +- `--agents N` — optional, default `4`. Cap at `6` (more invites + duplication and synthesis overhead). + +## Examples + +``` +# What this repo's biggest simplification wins are +/parallel-audit "simplification opportunities" + +# Test gaps before adding CI +/parallel-audit "test coverage gaps" --agents 3 + +# Pre-PR security pass +/parallel-audit "security review: subprocess, SQL strings, file paths" +``` + +## Instructions + +You are given `$ARGUMENTS`. Follow these steps. + +### 1. Parse args + +- First non-flag token sequence (until a `--` flag) is the `TOPIC`. +- `--agents N` sets the agent count; default `4`, clamp to `[2, 6]`. + +### 2. Pick the slicing dimensions + +Choose `N` **non-overlapping** slices that together cover the topic. +Slicing strategies (pick the one that fits the topic): + +- **By directory**: `bin/`, `utils/`, `data/`, `web/`, `tests/`, + `wiki/`. Best when the topic is breadth-first and mechanical. +- **By concern**: code quality / CLI ergonomics / data layer / + hygiene+CI / docs. Best for "code review" type asks. +- **By layer**: entry points / library / persistence / external IO. + Best for performance, security, error-handling. +- **By risk**: critical paths / hot paths / cold paths. Best for + performance and reliability audits. + +Reject overlap. If two slices would inspect the same files for the +same thing, merge them and pick a different fourth slice. + +### 3. Brief each agent + +Spawn all `N` agents in **a single message with N parallel tool +calls** (not sequentially). Each brief must include: + +- The slice's scope (paths, file globs, or symbol patterns). +- The exact question for this slice — narrow, not the umbrella topic. +- The output contract: + - Top ~8 findings, prioritized. + - Each finding: `file:line` + one-sentence problem + + one-sentence fix + severity tag `[critical|high|medium|low]`. + - Word cap (typically 500–600 words per agent). + - Final line: a one-sentence "biggest single win" recommendation. +- What to **skip** (the slices owned by sibling agents) — prevents + duplication. + +Do not delegate synthesis. Each agent reports raw findings; the +lead (you) ranks and dedupes. + +### 4. Synthesize + +Once all agents return: + +1. Collect findings, **dedupe** items mentioned by multiple agents + (these are higher-confidence — flag them). +2. Group by severity, then by theme. +3. Trim noise: cosmetic style, hypothetical "what-ifs", anything + without a file:line citation. +4. Verify any surprising claim before including it. Agent reports + can be wrong about file contents — when an agent's claim drives + a destructive recommendation, run a quick `Read`/`grep` first. +5. End with a clear "if you do one thing, do this" recommendation + and explicit ROI ordering. + +### 5. Report + +Produce the punch list in the conversation. Format: + +``` +## [Severity] — [Theme] +1. [file:line] [problem]. [Fix]. +2. ... + +## Biggest leverage if you do one thing +[One paragraph, named action, expected ROI.] + +Want me to start on [#N]? +``` + +Keep the synthesis tight — usually under 800 words. Long synthesis +defeats the point of fanning out. + +## Notes + +- The audit is **read-only**. Do not let agents edit files. The + Explore subagent type already enforces this; if you use a + general-purpose agent, say so explicitly in the brief. +- This skill is for **breadth**, not depth. For deep dives on a + specific bug, use a single agent with full context, not parallel + ones with partial context. +- Agents do not see the conversation history. Briefs must be + self-contained — no "based on what we discussed" references. +- The synthesis lives in the lead conversation, not in any agent. + Don't ask an agent to "synthesize the others' findings" — they + can't see them. +- If an agent returns "nothing notable", trust it. A clean slice + is information; don't pad the punch list to look thorough. +- For Mu2e prodtools specifically, the canonical 4-slice cut is: + `utils+bin code quality / CLI ergonomics+EXAMPLES.md drift / + DB schema+JSON configs / tests+repo hygiene`. Use this as the + default unless the topic argues otherwise. diff --git a/.claude/commands/poms-push.md b/.claude/commands/poms-push.md new file mode 100644 index 0000000..3704e41 --- /dev/null +++ b/.claude/commands/poms-push.md @@ -0,0 +1,171 @@ +--- +description: Plan a production POMS push — pick the right map number (extend existing in place vs allocate new) before delegating to /mu2epro-run +argument-hint: --dsconf [--workflow ] [SIMJOB_VERSION] +allowed-tools: Bash, Read +--- + +# Plan a production POMS push + +Decides whether a `--prod --jobdefs ` push should **extend an +existing POMS map in place** or **allocate a new map number**, based +on whether the workflow's earlier stages already live in a map. +Prints the recommended `/mu2epro-run` invocation; does **not** push. + +The PBI chain is the canonical example: stages 1+2 already in +`MDC2025-025.json` → stage 3 reco must extend `025`, not allocate +`026`. This skill exists because that mistake was made on +2026-04-25 and remediation required deleting an orphan SAM index; +the convention is now codified here. + +## Usage + +``` +/poms-push --dsconf [--workflow ] [SIMJOB_VERSION] +``` + +- `` — relative path to a prodtools JSON config (e.g. + `data/mdc2025/reco.json`); resolved against the repo root. +- `--dsconf ` — required; the dsconf `json2jobdef` will use + (e.g. `MDC2025ai_best_v1_3`). Used to (a) filter the JSON config + entries that will be pushed, and (b) forward to `/mu2epro-run`. +- `--workflow ` — optional substring or glob to match + against existing tarball descs (e.g. `PBI`, `PBI*_33344`, + `CeEndpoint`). If omitted, derived from the config's `desc` / + `tarball_append` fields. +- `SIMJOB_VERSION` — optional SimJob version tag (e.g. `MDC2025ai`, + `Run1Bag`); default `Run1Bag`. Forwarded to `/mu2epro-run`. + +## Examples + +``` +# Auto-derive workflow from data/mdc2025/reco.json desc fields +/poms-push data/mdc2025/reco.json --dsconf MDC2025ai_best_v1_3 MDC2025ai + +# Explicit workflow pattern (skip auto-detection) +/poms-push data/mdc2025/digi.json --dsconf MDC2025ai --workflow PBI MDC2025ai + +# Brand-new workflow (no prior stages in any map) +/poms-push data/mdc2025/stage1.json --dsconf MDC2025aj --workflow NewThing MDC2025aj +``` + +## Instructions + +You are given `$ARGUMENTS`. Follow these steps. + +### 1. Parse args + +- Locate `--dsconf `. **Error and stop** if missing. +- Locate `--workflow ` (optional). +- Locate a SimJob version tag — first positional token matching + `^[A-Z][A-Za-z0-9_]+$` and not a `--flag`; e.g. `MDC2025ai`, + `Run1Bag`. Default `Run1Bag`. +- The first remaining positional arg = `` (relative + path). Resolve it against the repo root (`cwd` at invocation + time) into an absolute path `JSON_ABS`. +- Validate `JSON_ABS` exists; error if not. + +### 2. Determine the workflow pattern + +If `--workflow ` was given, use it verbatim. Otherwise: + +a. Read `JSON_ABS`. The config is either a list of entries or a + single entry. Filter to only entries whose `dsconf` matches + `` (entries can hold `dsconf` as a string or a one-element + array). If no entries match, error and stop. + +b. From each filtered entry, extract a "workflow root" token: + - Prefer `desc` field if present. + - Else infer from `input_data` keys: parse `....`, + take the `desc` portion. + - Strip any of these stage-suffix patterns from the right end + (longest first): + `Mix1BBTriggered`, `Mix1BBTriggerable`, `Mix2BBTriggered`, + `Mix2BBTriggerable`, `Mix1BB-reco`, `Mix2BB-reco`, + `Mix1BB`, `Mix2BB`, `OnSpillTriggered`, `OffSpillTriggered`, + `OnSpill`, `OffSpill`, `Triggered`, `Triggerable`, `-reco`, + `Cat`. Iterate until none match. + - Also account for the entry's `tarball_append` (e.g. `-reco`) + by ensuring it's stripped. + +c. Take the **longest common prefix** of the resulting roots + (case-sensitive). Examples: + - `PBINormal_33344`, `PBIPathological_33344` → `PBI` + - `CeMLeadingLog`, `CePLeadingLog` → `Ce` + - `FlatGamma`, `FlatGammaCalo` → `FlatGamma` + +d. If the common prefix is shorter than 3 chars or empty, **stop + and ask** the user for `--workflow` explicitly. Don't guess. + +Print the derived pattern with one-line rationale. + +### 3. Scan existing maps + +- Run: `ls /exp/mu2e/app/users/mu2epro/production_manager/poms_map/MDC2025-*.json`. +- Keep only files whose basename matches `^MDC2025-\d{3}\.json$` + (exactly 3 digits). This excludes test variants (`-test`, + `-test2`, `-tes`), special names (`-MDS3c`), and any other + naming oddity in one stroke. +- For each remaining map, read it and extract every `tarball` value + (each entry has one). For each tarball name, parse out the desc + portion: `cnf.....tar` → ``. +- Count how many tarballs in each map have a `` that + **contains the workflow pattern as a substring** (case-sensitive). + +### 4. Decide the map + +- **Exactly one non-test map has ≥ 1 matching tarball** → + *extend in place*. Target = that map's path. +- **Multiple maps match** → stop and ask which one to extend. + Show each candidate with its match count and a sample tarball. +- **Zero maps match** → *allocate next free number*: + - Across all `MDC2025-NNN.json` matching `^MDC2025-\d{3}\.json$`, + take `max(NNN) + 1`, format as 3-digit zero-padded. + - Target = `/exp/mu2e/.../poms_map/MDC2025-.json` (does + not exist yet; `json2jobdef --prod --jobdefs ` will + create it). + +### 5. Report and stop + +Print exactly this shape: + +``` +Workflow pattern: (source: <--workflow flag | derived from desc>) +Existing maps: + MDC2025-NNN.json — matching tarballs (sample: ) + ... +Decision: +Reason: + +Recommended command: + + /mu2epro-run json2jobdef \ + --json \ + --dsconf \ + --prod \ + --jobdefs /exp/mu2e/app/users/mu2epro/production_manager/poms_map/MDC2025-NNN.json +``` + +**Do not invoke `/mu2epro-run` yourself.** Print and stop. The user +inspects the decision and runs the command next. This keeps the +existing production-push confirmation gate in `/mu2epro-run` exactly +where it is. + +## Notes + +- Read-only by design — does not modify maps, push tarballs, call + samweb, or change SAM definitions. +- `-test.json` maps are skipped intentionally. If a test-mode push + is needed, pass `--workflow ` explicitly and target + a `-test` map by hand. +- The longest-common-prefix heuristic can over-match (e.g. `Ce` + catches both `CeEndpoint` and `CeMLeadingLog` workflows). If the + derived pattern matches multiple unrelated maps, the skill stops + per step 4 and asks. Pass `--workflow` to disambiguate. +- If the JSON config has multiple `dsconf` values, only the entries + matching the supplied `--dsconf` are used to derive the pattern + (mirrors what `json2jobdef --dsconf` will do at push time). +- This skill encodes the convention in + `feedback_extend_existing_poms_map.md` (memory) and the Stage 3 + "Process note" subsection of + `wiki/pages/pbi-sequence-workflow.md`. If the convention changes, + update both alongside this skill. diff --git a/.claude/commands/recent-datasets.md b/.claude/commands/recent-datasets.md new file mode 100644 index 0000000..fb3bc61 --- /dev/null +++ b/.claude/commands/recent-datasets.md @@ -0,0 +1,109 @@ +--- +description: List recently created datasets with completeness — sources Mu2e env + pyenv ana, defaults to last 1 day with --completeness column +argument-hint: [days] [--query ] [extra listNewDatasets flags] +allowed-tools: Bash +--- + +# List recent datasets with completeness + +Thin wrapper over `bin/listNewDatasets --completeness` that does all +the env setup so you can ask "what landed recently and is it +complete?" with one command. Encodes: + +- `source setupmu2e-art.sh && muse setup ops && pyenv ana` + (the last is required for SQLAlchemy; without it + `--completeness` degrades to a warning) +- `python3 bin/listNewDatasets` (not `bash` — the wrapper has a + Python shebang) +- `--completeness` flag on by default (auto-rebuilds the POMS DB + if any map is newer than the DB, within the lookback window) +- `--days 1` by default (more useful than the 7-day default for + "what changed today") +- Output filtered to drop the noisy `Skipping logparser ...` + rebuild trace lines + +## Usage + +``` +/recent-datasets [days] [--query ] [extra-args] +``` + +- `[days]` — optional first positional integer, sets `--days N`. + Default `1`. Also controls the DB-staleness lookback window. +- `--query ` — pass-through to `listNewDatasets --query`, + for SAM where-clauses (e.g. `"dh.dataset like 'mcs.mu2e.PBI%'"`). + When given, `--days` only governs DB staleness, not the SAM + filter (the custom query overrides the date filter). +- Anything else — passed through verbatim + (`--user oksuzian`, `--filetype log`, `--no-rebuild`, + `--size`, etc.). + +## Examples + +``` +# What landed in the last day, with completeness +/recent-datasets + +# Last 7 days +/recent-datasets 7 + +# All recent PBI mcs files (regardless of date — query overrides date) +/recent-datasets --query "dh.dataset like 'mcs.mu2e.PBI%Mix1BB.MDC2025ai_best_v1_3.art'" + +# Last day, your own datasets, with file sizes +/recent-datasets 1 --user oksuzian --size + +# Skip the auto-rebuild even if DB is stale +/recent-datasets 7 --no-rebuild +``` + +## Instructions + +You are given `$ARGUMENTS`. Follow these steps. + +### 1. Parse args + +- If the first whitespace-separated token is a positive integer, + treat it as `DAYS` and drop it from the argv. Otherwise + `DAYS=1`. +- Everything else is `EXTRA_ARGS` (passed through). + +### 2. Resolve repo root + +Set `REPO=$PWD` at invocation time. The wrapper at +`$REPO/bin/listNewDatasets` is the entry point. + +### 3. Run + +Execute as a single Bash command so the sourced env is live for +the listNewDatasets call: + +```bash +source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh > /dev/null 2>&1 \ + && muse setup ops > /dev/null 2>&1 \ + && pyenv ana > /dev/null 2>&1 \ + && python3 /bin/listNewDatasets --completeness --days 2>&1 \ + | grep -v -E '^(Skipping logparser|Loading [0-9]+ JSON files|Loaded [0-9]+ job definitions|Removed [0-9]+ jobs|Computing completion status|Marked [0-9]+ jobs as complete|Discovered and cached [0-9]+ derived datasets|Error listing definition files|Error describing definition|Warning: Could not count files| Template mode:|^$)' +``` + +### 4. Report + +Print the filtered output to the user. The table that survives the +filter — header, dividers, dataset rows, completeness column — is +what they actually want. The DB staleness/rebuild messages survive +the filter on purpose: the user should know if a slow rebuild +happened. + +## Notes + +- The filter is heuristic; if a future `listNewDatasets` / + `db_builder` change introduces new noise lines, add their + prefixes to the grep. If a real warning gets accidentally + filtered, drop the matching pattern. +- Read-only by design — no SAM writes, no DB rebuild beyond what + `listNewDatasets --completeness` already does internally. +- For "what's *not yet* in production", use `pomsMonitor + --campaign --outputs --incomplete` directly instead; + this skill is for the "what landed in SAM" angle. +- For per-dataset family trees use `famtree`; for per-dataset + log metrics use `logparser`. This skill is intentionally narrow. diff --git a/.gitignore b/.gitignore index 29e2b75..d973121 100644 --- a/.gitignore +++ b/.gitignore @@ -110,3 +110,17 @@ prodtools_cvmfs/ indir/ web/mermaid/ web/*.md + +# Local POMS DB (derived from poms_map/, rebuildable) +poms_data.db + +# Scratch / local notes / test workspaces (root only) +/os +/sys +/prompts*.txt +/mu2e_common.gdml +/MDC*-test.json +/momentum_resolution_*.png +/test2/ +/test_runmu2e/ +/test_reco/ diff --git a/CLAUDE.md b/CLAUDE.md index 0aefed0..26487e3 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,7 @@ Before answering any question about running the prodtools commands (`json2jobdef`, `jobfcl`, `fcldump`, `runmu2e`, `jobdef`, `jsonexpander`, `jobquery`, `mkidxdef`, `pomsMonitor`, `famtree`, `logparser`, `genFilterEff`, `datasetFileList`, `listNewDatasets`, `mkrecovery`, -`copy_to_stash`, `listRelatedDefs`), read `EXAMPLES.md` at the repo +`copy_to_stash`), read `EXAMPLES.md` at the repo root. It is the authoritative reference for CLI flags, JSON config shapes, and canonical invocations. Do not guess flags or copy patterns from memory — consult the current doc. diff --git a/bin/add_inputs_from_list.py b/bin/add_inputs_from_list.py new file mode 100644 index 0000000..70a4f99 --- /dev/null +++ b/bin/add_inputs_from_list.py @@ -0,0 +1,52 @@ +#!/usr/bin/env python3 +import argparse +import json +import os +import sys + + +def main() -> None: + parser = argparse.ArgumentParser(description="Append dataset names from a list into input_data of evntuple.json") + parser.add_argument("list_file", help="Path to text file with dataset names (one per line)") + parser.add_argument( + "--json", + dest="json_path", + default=os.path.join("data", "mdc2020", "evntuple.json"), + help="Path to evntuple.json (default: data/mdc2020/evntuple.json)", + ) + args = parser.parse_args() + + # Read JSON file + with open(args.json_path, "r") as f: + data = json.load(f) + + # Get existing dataset names (avoid duplicates) + existing = {next(iter(item.keys())) for item in data[0]["input_data"]} + + # Read new datasets from text file + with open(args.list_file, "r") as f: + new_datasets = [line.strip() for line in f if line.strip() and not line.strip().startswith("#")] + + # Add only new datasets + added = 0 + for dataset in new_datasets: + if dataset not in existing: + data[0]["input_data"].append({dataset: 1}) + existing.add(dataset) + added += 1 + + if added == 0: + print("No new items to add (all already present)") + return + + # Write back JSON file + with open(args.json_path, "w") as f: + json.dump(data, f, indent=2) + + print(f"Added {added} item(s) to {args.json_path}") + + +if __name__ == "__main__": + main() + + diff --git a/bin/install_prodtools.sh b/bin/install_prodtools.sh new file mode 100755 index 0000000..b608c48 --- /dev/null +++ b/bin/install_prodtools.sh @@ -0,0 +1,111 @@ +#!/bin/bash +# install_prodtools.sh - Install a prodtools release on CVMFS +# Run directly on cvmfsmu2e@oasiscfs.fnal.gov +# Usage: ./install_prodtools.sh [-n] [-t [DIR]] v1.6.3 +# -n Dry run: check GitHub tag and install path, but make no changes. +# -t [DIR] Test mode: skip cvmfs_server calls and install into a local +# writable dir (default: a fresh mktemp -d). Lets you exercise +# the fetch/extract/mv/symlink logic without touching CVMFS. + +set -e + +DRY_RUN=false +TEST_MODE=false +TEST_BASE="" +while [[ "$1" == -* ]]; do + case "$1" in + -n) DRY_RUN=true; shift ;; + -t) + TEST_MODE=true + shift + # Optional path argument for -t. Anything starting with '-' or nothing + # left = default to mktemp. + if [[ -n "$1" && "$1" != -* && ! "$1" =~ ^v?[0-9]+\.[0-9]+ ]]; then + TEST_BASE="$1" + shift + fi + ;; + *) break ;; + esac +done + +VER=${1:?Usage: $0 [-n] [-t [DIR]] (e.g. v1.6.3)} +CVMFS_REPO=mu2e.opensciencegrid.org +if $TEST_MODE; then + INSTALL_BASE=${TEST_BASE:-$(mktemp -d /tmp/test_cvmfs_install.XXXXXX)} + mkdir -p "${INSTALL_BASE}" + echo "=== TEST MODE: install base is ${INSTALL_BASE} (no cvmfs_server calls) ===" +else + INSTALL_BASE=/cvmfs/${CVMFS_REPO}/bin/prodtools +fi +TMPDIR=$(mktemp -d) + +cleanup() { + rm -rf "${TMPDIR}" +} +trap cleanup EXIT + +abort_transaction() { + if $TEST_MODE; then + echo "ERROR: aborting (test mode — no cvmfs_server abort)" + return + fi + echo "ERROR: Aborting CVMFS transaction..." + cd ~ + cvmfs_server abort -f ${CVMFS_REPO} 2>/dev/null || true + echo "WARNING: Per CVMFS policy, run an empty transaction+publish to restore correct permissions:" + echo " cvmfs_server transaction ${CVMFS_REPO} && cvmfs_server publish ${CVMFS_REPO}" +} +trap abort_transaction ERR + +# Verify the GitHub release tag exists. +# -L is required: github.com archive URLs 302-redirect to S3, and without +# -L a bare HEAD returns the redirect as a successful 3xx even when the +# tag doesn't exist (false positive). With -L we follow through to the +# real resource and fail cleanly on 404. +echo "Checking GitHub release ${VER}..." +curl -fsIL "https://github.com/Mu2e/prodtools/archive/refs/tags/${VER}.tar.gz" > /dev/null \ + || { echo "ERROR: Release ${VER} not found on GitHub."; exit 1; } +echo "Release ${VER} found." + +# Check not already installed +if [ -d "${INSTALL_BASE}/${VER}" ]; then + echo "ERROR: ${INSTALL_BASE}/${VER} already exists. Aborting." + exit 1 +fi + +if $DRY_RUN; then + echo "[dry-run] Would install to ${INSTALL_BASE}/${VER}" + echo "[dry-run] Would update ${INSTALL_BASE}/current -> ${VER}" + echo "[dry-run] No changes made." + exit 0 +fi + +if $TEST_MODE; then + echo "Skipping cvmfs_server transaction (test mode)" +else + echo "Opening CVMFS transaction..." + cvmfs_server transaction ${CVMFS_REPO} +fi + +echo "Downloading and extracting prodtools ${VER}..." +curl -fsSL "https://github.com/Mu2e/prodtools/archive/refs/tags/${VER}.tar.gz" \ + | tar -xz -C ${TMPDIR} + +# GitHub strips the leading 'v' from the directory name +SRC_DIR=$(ls -d ${TMPDIR}/prodtools-*/) +mv "${SRC_DIR}" "${INSTALL_BASE}/${VER}" + +echo "Updating 'current' symlink to ${VER}..." +ln -sfn "${VER}" "${INSTALL_BASE}/current" + +if $TEST_MODE; then + echo "Skipping cvmfs_server publish (test mode)" +else + echo "Publishing to CVMFS..." + cd ~ + cvmfs_server publish ${CVMFS_REPO} +fi + +echo "Done. prodtools ${VER} is now available at ${INSTALL_BASE}/${VER}" +echo " 'current' symlink points to ${VER}" diff --git a/bin/list_no_child_datasets b/bin/list_no_child_datasets new file mode 100755 index 0000000..f5a3d3c --- /dev/null +++ b/bin/list_no_child_datasets @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +"""List all datasets without children and complete datasets from the database.""" + +import os +import sys + +# Add parent directory to path for imports +parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, parent_dir) + +try: + import sqlalchemy # noqa: F401 +except ImportError: + sys.stderr.write("error: SQLAlchemy not found. Run 'pyenv ana' after 'muse setup ops'.\n") + sys.exit(2) + +from utils.poms_db import get_db_session, DatasetInfo, JobOutput, Job +from utils.db_analyzer import get_default_db_path + +def main(): + session = get_db_session(get_default_db_path()) + + # Query datasets that are both without children AND outputs of complete jobs + datasets = session.query(DatasetInfo).join( + JobOutput, DatasetInfo.dataset_name == JobOutput.dataset + ).join( + Job, JobOutput.job_id == Job.id + ).filter( + DatasetInfo.has_children == False, + Job.complete == True + ).distinct().all() + + if not datasets: + print("No datasets found (no children AND complete).") + return + + print(f"Found {len(datasets)} datasets (no children AND complete):\n") + for info in sorted(datasets, key=lambda x: x.dataset_name): + print(info.dataset_name) + +if __name__ == '__main__': + main() + diff --git a/bin/plot_straw_hits.py b/bin/plot_straw_hits.py new file mode 100755 index 0000000..e3cbc62 --- /dev/null +++ b/bin/plot_straw_hits.py @@ -0,0 +1,252 @@ +#!/usr/bin/env python3 +""" +Plot Mu2e tracker straw hit rates from an EventNtuple ROOT file. + +Produces: + hits_per_plane.png -- hit count vs plane number (0-35) + hits_per_panel.png -- hit count vs panel number within plane (0-5) + hits_per_straw.png -- hit count vs straw number within panel (0-95) + occupancy_plane_panel.png -- 2D occupancy: plane x panel + occupancy_panel_straw.png -- 2D occupancy: absolute panel index x straw + hits_per_plane_overlay.png -- all hits vs active hits per plane + +Requires: + pip install git+https://github.com/Mu2e/pyutils.git + +Usage: + python plot_straw_hits.py [options] + + # typical for from_dig-OnSpill output: + python plot_straw_hits.py nts.owner.trkana-triggerMC.version.sequencer.root \\ + --tree EventNtupleTTMCTpr/ntuple + + # typical for standard reco output: + python plot_straw_hits.py nts.mu2e.*.root \\ + --tree EventNtuple/ntuple +""" + +import argparse +import os +import sys + +import awkward as ak +import numpy as np + +from pyutils.pyprocess import Processor +from pyutils.pyplot import Plot + +# --------------------------------------------------------------------------- +# Mu2e tracker geometry constants +# --------------------------------------------------------------------------- +N_PLANES = 36 # planes in the tracker +N_PANELS = 6 # panels per plane +N_STRAWS = 96 # straws per panel + + +# --------------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------------- +def parse_args(): + p = argparse.ArgumentParser( + description="Plot Mu2e tracker straw hit rates using pyutils", + formatter_class=argparse.ArgumentDefaultsHelpFormatter, + ) + p.add_argument("file", help="EventNtuple ROOT file (local path)") + p.add_argument( + "--tree", + default="EventNtupleTTMCTpr/ntuple", + help="TTree path inside the ROOT file", + ) + p.add_argument( + "--outdir", default=".", help="Output directory for PNG plots" + ) + return p.parse_args() + + +# --------------------------------------------------------------------------- +# Data loading +# --------------------------------------------------------------------------- +def load_data(file_name: str, tree_path: str) -> dict: + processor = Processor( + tree_path=tree_path, + use_remote=False, + verbosity=1, + ) + data = processor.process_data( + file_name=file_name, + branches=["trkhits"], + ) + return data + + +# --------------------------------------------------------------------------- +# Helper: flatten a per-track-hit field to a 1-D numpy array +# --------------------------------------------------------------------------- +def flat(hits, field: str): + return ak.to_numpy(ak.flatten(hits[field], axis=None)) + + +# --------------------------------------------------------------------------- +# Plots +# --------------------------------------------------------------------------- +def plot_hits_per_plane(hits_all, hits_active, plotter: Plot, outdir: str): + """1-D: all hits and active-only hits per plane, overlaid.""" + plane_all = flat(hits_all, "plane") + plane_active = flat(hits_active, "plane") + + plotter.plot_1D_overlay( + {"All hits": plane_all, "Active hits": plane_active}, + nbins=N_PLANES, + xmin=-0.5, + xmax=N_PLANES - 0.5, + title="Straw Hits per Plane", + xlabel="Plane number", + ylabel="Hit count", + out_path=os.path.join(outdir, "hits_per_plane.png"), + show=False, + ) + print(" -> hits_per_plane.png") + + +def plot_hits_per_panel(hits_all, hits_active, plotter: Plot, outdir: str): + """1-D: hit count per panel within plane, all and active overlaid.""" + panel_all = flat(hits_all, "panel") + panel_active = flat(hits_active, "panel") + + plotter.plot_1D_overlay( + {"All hits": panel_all, "Active hits": panel_active}, + nbins=N_PANELS, + xmin=-0.5, + xmax=N_PANELS - 0.5, + title="Straw Hits per Panel (within plane)", + xlabel="Panel number", + ylabel="Hit count", + out_path=os.path.join(outdir, "hits_per_panel.png"), + show=False, + ) + print(" -> hits_per_panel.png") + + +def plot_hits_per_straw(hits_all, hits_active, plotter: Plot, outdir: str): + """1-D: hit count per straw within panel, all and active overlaid.""" + straw_all = flat(hits_all, "straw") + straw_active = flat(hits_active, "straw") + + plotter.plot_1D_overlay( + {"All hits": straw_all, "Active hits": straw_active}, + nbins=N_STRAWS, + xmin=-0.5, + xmax=N_STRAWS - 0.5, + title="Straw Hits per Straw (within panel)", + xlabel="Straw number", + ylabel="Hit count", + out_path=os.path.join(outdir, "hits_per_straw.png"), + show=False, + ) + print(" -> hits_per_straw.png") + + +def plot_occupancy_plane_panel(hits, plotter: Plot, outdir: str, tag: str = ""): + """2-D occupancy: plane (x) vs panel (y).""" + plane = flat(hits, "plane") + panel = flat(hits, "panel") + + label = " (active only)" if tag else "" + fname = f"occupancy_plane_panel{tag}.png" + plotter.plot_2D( + plane, + panel, + nbins_x=N_PLANES, + nbins_y=N_PANELS, + xmin=-0.5, + xmax=N_PLANES - 0.5, + ymin=-0.5, + ymax=N_PANELS - 0.5, + title=f"Straw Hit Occupancy{label}", + xlabel="Plane number", + ylabel="Panel number", + cmap="inferno", + out_path=os.path.join(outdir, fname), + show=False, + ) + print(f" -> {fname}") + + +def plot_occupancy_panel_straw(hits, plotter: Plot, outdir: str, tag: str = ""): + """2-D occupancy: absolute panel index (x) vs straw (y). + + Absolute panel index = plane * N_PANELS + panel (range 0-215). + """ + plane = flat(hits, "plane") + panel = flat(hits, "panel") + straw = flat(hits, "straw") + abs_panel = plane * N_PANELS + panel + + label = " (active only)" if tag else "" + fname = f"occupancy_panel_straw{tag}.png" + plotter.plot_2D( + abs_panel, + straw, + nbins_x=N_PLANES * N_PANELS, + nbins_y=N_STRAWS, + xmin=-0.5, + xmax=N_PLANES * N_PANELS - 0.5, + ymin=-0.5, + ymax=N_STRAWS - 0.5, + title=f"Straw Occupancy Map{label}", + xlabel="Absolute panel index (plane\u00d76 + panel)", + ylabel="Straw number", + cmap="inferno", + out_path=os.path.join(outdir, fname), + show=False, + ) + print(f" -> {fname}") + + +# --------------------------------------------------------------------------- +# Main +# --------------------------------------------------------------------------- +def main(): + args = parse_args() + + os.makedirs(args.outdir, exist_ok=True) + + print(f"\nLoading : {args.file}") + print(f"Tree path: {args.tree}") + + data = load_data(args.file, args.tree) + + # Split into all-hits and active-only sub-collections + hits_all = data["trkhits"] + hits_active = hits_all[hits_all["dactive"]] + + n_all = int(ak.sum(ak.num(hits_all["plane"]))) + n_active = int(ak.sum(ak.num(hits_active["plane"]))) + n_tracks = int(ak.sum(ak.num(data["trkhits"]["plane"]) >= 0)) + + print(f"\nTracks : {n_tracks}") + print(f"All hits : {n_all}") + print(f"Active : {n_active} ({100*n_active/max(n_all,1):.1f} %)") + + if n_all == 0: + print("\nNo hits found — check that the tree path is correct and the") + print("trigger SelectEvents condition was satisfied in this file.") + sys.exit(0) + + plotter = Plot(verbosity=0) + + print(f"\nWriting plots to: {os.path.abspath(args.outdir)}/") + + plot_hits_per_plane(hits_all, hits_active, plotter, args.outdir) + plot_hits_per_panel(hits_all, hits_active, plotter, args.outdir) + plot_hits_per_straw(hits_all, hits_active, plotter, args.outdir) + plot_occupancy_plane_panel(hits_all, plotter, args.outdir, tag="") + plot_occupancy_plane_panel(hits_active, plotter, args.outdir, tag="_active") + plot_occupancy_panel_straw(hits_all, plotter, args.outdir, tag="") + plot_occupancy_panel_straw(hits_active, plotter, args.outdir, tag="_active") + + print("\nDone.") + + +if __name__ == "__main__": + main() diff --git a/bin/pomsMonitor b/bin/pomsMonitor index 04a8738..07528ec 100755 --- a/bin/pomsMonitor +++ b/bin/pomsMonitor @@ -8,6 +8,12 @@ import sys parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, parent_dir) +try: + import sqlalchemy # noqa: F401 +except ImportError: + sys.stderr.write("error: SQLAlchemy not found. Run 'pyenv ana' after 'muse setup ops'.\n") + sys.exit(2) + from utils.pomsMonitor import main if __name__ == '__main__': diff --git a/bin/pomsMonitorWeb b/bin/pomsMonitorWeb index 4f32ca2..b251036 100755 --- a/bin/pomsMonitorWeb +++ b/bin/pomsMonitorWeb @@ -9,6 +9,21 @@ import shlex parent_dir = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) sys.path.insert(0, parent_dir) +_missing = [] +try: + import sqlalchemy # noqa: F401 +except ImportError: + _missing.append("SQLAlchemy") +try: + import flask # noqa: F401 +except ImportError: + _missing.append("Flask") +if _missing: + sys.stderr.write( + f"error: {' and '.join(_missing)} not found. Run 'pyenv ana' after 'muse setup ops'.\n" + ) + sys.exit(2) + from flask import Flask, jsonify, request, redirect from utils.poms_db import get_db_session, Job, DatasetInfo from utils.db_analyzer import get_default_db_path diff --git a/bin/setup_run1b.sh b/bin/setup_run1b.sh new file mode 100644 index 0000000..74f78e4 --- /dev/null +++ b/bin/setup_run1b.sh @@ -0,0 +1,14 @@ +#!/bin/bash +# Lightweight helper to prepare environment for Run1B CeEndpoint digitization +# Usage: source bin/setup_run1b.sh + +# Source the project-wide setup (if present) +if [ -f "$(dirname "$BASH_SOURCE")/setup.sh" ]; then + source "$(dirname "$BASH_SOURCE")/setup.sh" +fi + +# Ensure local directory is in MU2E_SEARCH_PATH so fcl/ files are discoverable +export MU2E_SEARCH_PATH=".:$MU2E_SEARCH_PATH" + +# Echo what we did for convenience +echo "Sourced setup.sh (if present) and set MU2E_SEARCH_PATH to include ." diff --git a/data/Run1B/digi.json b/data/Run1B/digi.json new file mode 100644 index 0000000..ec589ab --- /dev/null +++ b/data/Run1B/digi.json @@ -0,0 +1,83 @@ +[ + { + "dsconf": "Run1Bab2_best_v1_2", + "fcl": "Production/JobConfig/digitize/OnSpill.fcl", + "input_data": [ + {"dts.mu2e.CeEndpoint.Run1Bab.art": 100}, + {"dts.mu2e.DIOtail0_60.Run1Bab.art": 100}, + {"dts.mu2e.DIOtail60_80.Run1Bab.art": 100}, + {"dts.mu2e.DIOtail80_90.Run1Bab.art": 100}, + {"dts.mu2e.DIOtail90_inf.Run1Bab.art": 100}, + {"dts.mu2e.FlatGamma.Run1Bab.art": 100}, + {"dts.mu2e.FlateMinus.Run1Bab.art": 100} + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_2", + "outputs.TriggeredOutput.fileName": "dig.owner.{desc}OnSpillTriggered.version.sequencer.art", + "outputs.TriggerableOutput.fileName": "dig.owner.{desc}OnSpillTriggerable.version.sequencer.art", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "outputs.TriggerableOutput.SelectEvents" : [] + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bab/setup.sh" + }, + { + "dsconf": "Run1Bah_best_v1_4-000", + "fcl": "Production/JobConfig/digitize/OnSpill.fcl", + "input_data": [ + {"dts.mu2e.CeEndpoint.Run1Bag.art": 100}, + {"dts.mu2e.DIOtail0_60.Run1Bag.art": 100}, + {"dts.mu2e.DIOtail60_80.Run1Bag.art": 100}, + {"dts.mu2e.DIOtail80_90.Run1Bag.art": 100}, + {"dts.mu2e.DIOtail90_inf.Run1Bag.art": 100}, + {"dts.mu2e.FlatGamma.Run1Bag.art": 100}, + {"dts.mu2e.FlateMinus.Run1Bag.art": 100}, + {"dts.mu2e.CosmicCRYAll.Run1Bah.art": 100}, + {"dts.mu2e.RPCExternal.Run1Bah.art": 100} + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "#include": [ + "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl" + ] + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh" + }, + { + "dsconf": "Run1Bah_best_v1_4-000", + "fcl": "Production/JobConfig/digitize/OnSpill.fcl", + "input_data": [ + {"dts.mu2e.MuCap1809keVCalo.Run1Bag.art": 100} + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "outputs.Output.fastCloning": "false", + "#include": [ + "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "Production/JobConfig/mixing/NoPrimary.fcl" + ] + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh" + } +] diff --git a/data/Run1B/evntuple.json b/data/Run1B/evntuple.json new file mode 100644 index 0000000..98085e0 --- /dev/null +++ b/data/Run1B/evntuple.json @@ -0,0 +1,197 @@ +[ + { + "dsconf": ["MDC2025-000"], + "fcl": ["EventNtuple/fcl/from_mcs-extracted.fcl"], + "input_data": [ + {"mcs.mu2e.CeEndpointOnSpillTriggerableTriggerable-KL.Run1Bab2_best_v1_2.art": 2}, + {"mcs.mu2e.DIOOnSpillTriggerableTriggerable-KL.Run1Bab2_best_v1_2.art": 1}, + {"mcs.mu2e.DIOtail50OnSpillTriggerableTriggerable-KL.Run1Bab2_best_v1_2.art": 2}, + {"mcs.mu2e.DIOtail80OnSpillTriggerableTriggerable-KL.Run1Bab2_best_v1_2.art": 5}, + {"mcs.mu2e.DIOtail90OnSpillTriggerableTriggerable-KL.Run1Bab2_best_v1_2.art": 5} + ], + "fcl_overrides": [ + { + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root" + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_01_01/setup.sh"] + }, + { + "dsconf": ["Run1B-003"], + "fcl": ["EventNtuple/fcl/from_mcs-Run1B.fcl"], + "input_data": [ + {"mcs.mu2e.CeEndpointMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.CeEndpointOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail0_60MixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail0_60OnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail60_80MixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail60_80OnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail80_90MixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail80_90OnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail90_infMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail90_infOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlatGammaMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlatGammaOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlateMinusMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlateMinusOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.NoPrimaryMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1} + ], + "fcl_overrides": [ + { + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root" + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_01_04/setup.sh"] + }, + { + "dsconf": ["Run1B-003"], + "fcl": ["EventNtuple/fcl/from_mcs-Run1B.fcl"], + "input_data": [ + {"mcs.mu2e.CeEndpointMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.CeEndpointOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail0_60MixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail0_60OnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail60_80MixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail60_80OnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail80_90MixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail80_90OnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail90_infMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.DIOtail90_infOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlatGammaMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlatGammaOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlateMinusMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.FlateMinusOnSpillTriggerable-KL.Run1Baf_best_v1_4-000.art": 1}, + {"mcs.mu2e.NoPrimaryMixLowTriggerable-KL.Run1Baf_best_v1_4-000.art": 1} + ], + "fcl_overrides": [ + { + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root" + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_01_04/setup.sh"] + }, + { + "dsconf": ["Run1B-004"], + "fcl": ["EventNtuple/fcl/from_mcs-Run1B.fcl"], + "input_data": [ + {"mcs.mu2e.CeEndpointMixLow-KL.Run1Baf_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail0_60MixLow-KL.Run1Baf_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail60_80MixLow-KL.Run1Baf_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail80_90MixLow-KL.Run1Baf_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail90_infMixLow-KL.Run1Baf_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlatGammaMixLow-KL.Run1Baf_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlateMinusMixLow-KL.Run1Baf_best_v1_4-001.art": 1}, + {"mcs.mu2e.NoPrimaryMixLow-KL.Run1Baf_best_v1_4-001.art": 1} + ], + "fcl_overrides": [ + { + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root" + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_01_04/setup.sh"] + }, + { + "dsconf": ["Run1B-005"], + "fcl": ["EventNtuple/fcl/from_mcs-Run1B.fcl"], + "input_data": [ + {"mcs.mu2e.CeEndpointMix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail0_60Mix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail60_80Mix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail80_90Mix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail90_infMix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlatGammaMix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlateMinusMix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.NoPrimaryMix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.MuCap1809keVCaloMix1BB-KL.Run1Bah_best_v1_4-001.art": 1}, + + {"mcs.mu2e.CeEndpoint-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail0_60-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail60_80-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail80_90-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.DIOtail90_inf-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlatGamma-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlateMinus-KL.Run1Bah_best_v1_4-001.art": 1}, + {"mcs.mu2e.MuCap1809keVCalo-KL.Run1Bah_best_v1_4-001.art": 1} + ], + "fcl_overrides": [ + { + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root" + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_02_00/setup.sh"] + }, + { + "dsconf": ["Run1B-007"], + "fcl": ["EventNtuple/fcl/from_mcs-Run1B.fcl"], + "input_data": [ + {"mcs.mu2e.NoPrimaryMix1BB-KL.Run1Bah_best_v1_4-002.art": 1} + ], + "fcl_overrides": [ + { + "physics.EventNtupleEndPath": [ "EventNtuple" ], + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root" + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_02_00/setup.sh"] + }, + { + "dsconf": ["Run1B-008"], + "fcl": ["EventNtuple/fcl/from_mcs-Run1B.fcl"], + "input_data": [ + {"mcs.mu2e.CeEndpointMix1BB-KL.Run1Bai_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlatGammaMix1BB-KL.Run1Bai_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlateMinusMix1BB-KL.Run1Bai_best_v1_4-001.art": 1}, + {"mcs.mu2e.NoPrimaryMix1BB-KL.Run1Bai_best_v1_4-001.art": 1}, + {"mcs.mu2e.MuCap1809keVCaloMix1BB-KL.Run1Bai_best_v1_4-001.art": 1}, + + {"mcs.mu2e.CeEndpoint-KL.Run1Bai_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlatGamma-KL.Run1Bai_best_v1_4-001.art": 1}, + {"mcs.mu2e.FlateMinus-KL.Run1Bai_best_v1_4-001.art": 1}, + {"mcs.mu2e.MuCap1809keVCalo-KL.Run1Bai_best_v1_4-001.art": 1} + ], + "fcl_overrides": [ + { + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root" + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_02_00/setup.sh"] + }, + { + "dsconf": ["Run1B-009"], + "fcl": ["EventNtuple/fcl/from_mcs-Run1B.fcl"], + "input_data": [ + {"mcs.mu2e.NoPrimaryMix1BB-KL.Run1Bai_best_v1_4-001.art": 1} + ], + "fcl_overrides": [ + { + "services.TFileService.fileName": "nts.mu2e.{desc}.version.sequencer.root", + "physics.EventNtupleEndPath": [ "EventNtuple" ] + } + ], + + "inloc": ["tape"], + "outloc": [{"*.root": "tape"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/AnalysisMDC2025/v01_02_00/setup.sh"] + } +] \ No newline at end of file diff --git a/data/Run1B/merge_beam_pileup.json b/data/Run1B/merge_beam_pileup.json new file mode 100644 index 0000000..0cba9ba --- /dev/null +++ b/data/Run1B/merge_beam_pileup.json @@ -0,0 +1,72 @@ +[ + { + "desc": "EleBeamFlashCat", + "dsconf": "Run1Bai-007", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.EleBeamFlash.Run1Bai-007.art": 100}, + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.EleBeamFlashCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + }, + { + "desc": "MuBeamFlashCat", + "dsconf": "Run1Bai-007", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.MuBeamFlash.Run1Bai-007.art": 100}, + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.MuBeamFlashCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + }, + { + "desc": "MuStopPileupCat", + "dsconf": "Run1Bai-007", + "input_data": {"dts.mu2e.MuStopPileup.Run1Bai-007.art": 100}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.MuStopPileupCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + }, + { + "desc": "NeutralsFlashCat", + "dsconf": "Run1Bai-003", + "input_data": {"dts.mu2e.NeutralsFlash.Run1Bai-003.art": 5000}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.NeutralsFlashCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + }, + { + "desc": "TargetStopsCat", + "dsconf": "Run1Bai-003", + "input_data": {"sim.mu2e.TargetStops.Run1Bai-003.art": 2500}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "sim.owner.TargetStopsCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + } +] diff --git a/data/Run1B/merge_filter.json b/data/Run1B/merge_filter.json new file mode 100644 index 0000000..073b1da --- /dev/null +++ b/data/Run1B/merge_filter.json @@ -0,0 +1,265 @@ +[ + { + "desc": "CosmicCRYSpillSplitter", + "dsconf": "MDC2025ac", + "fcl": "Production/JobConfig/cosmic/SpillSplitter.fcl", + "input_data": {"dts.mu2e.CosmicCRYAll.MDC2025ac.art": 10}, + "inloc": "tape", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh" + }, + { + "desc": "PiBeamCat", + "dsconf": "Run1Bah", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"sim.mu2e.PiBeam.Run1Bah.art": 100}, + "fcl_overrides": { + "outputs.out.fileName": "sim.owner.PiBeamCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh" + }, + { + "desc": "MuonIPAStopSelector", + "dsconf": "MDC2025ac", + "fcl": "Production/JobConfig/pileup/MuonIPAStopSelector.fcl", + "input_data": {"sim.mu2e.IPAStopsCat.MDC2025ac.art": 1}, + "inloc": "tape", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh" + }, + { + "desc": "EleBeamFlashCat", + "dsconf": "Run1Baa2", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.EleBeamFlash.Run1Baa.art": 4998}, + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.EleBeamFlashCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh" + }, + { + "desc": "EleBeamFlashCat", + "dsconf": "Run1Bag", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.EleBeamFlash.Run1Bag.art": 50}, + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.EleBeamFlashCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh" + }, + { + "desc": "MuBeamFlashCat", + "dsconf": "Run1Baa1", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.MuBeamFlash.Run1Baa1.art": 500}, + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.MuBeamFlashCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh" + }, + { + "desc": "MuBeamFlashCat", + "dsconf": "Run1Bag", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.MuBeamFlash.Run1Bag.art": 500}, + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.MuBeamFlashCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh" + }, + { + "desc": "NeutralsFlashCat", + "dsconf": "MDC2025ac", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.NeutralsFlash.MDC2025ac.art": 1000}, + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.NeutralsFlashCat.version.sequence.art" + }, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh" + }, + { + "desc": "MuStopPileupCat", + "dsconf": "MDC2025ac", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": {"dts.mu2e.MuStopPileup.MDC2025ac.art": 100}, + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.MuStopPileupCat.version.sequence.art" + }, + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh" + }, + { + "dsconf": "MDC2025ac", + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": [ + {"dts.mu2e.EarlyNeutralsFlash.MDC2025ac.art": 10000}, + {"dts.mu2e.EarlyMuBeamFlash.MDC2025ac.art": 10000}, + {"dts.mu2e.EarlyEleBeamFlash.MDC2025ac.art": 10000} + ], + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.{desc}Cat.version.sequence.art" + }, + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh" + }, + { + "desc": "BeamSplitter", + "dsconf": "MDC2025ae", + "fcl": "Production/JobConfig/beam/BeamSplitter.fcl", + "input_data": {"sim.mu2e.Beam.MDC2025ae.art": 5000}, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ae/setup.sh" + }, + { + "desc": "BeamSplitter", + "dsconf": "MDC2025ac", + "fcl": "Production/JobConfig/beam/BeamSplitter.fcl", + "input_data": {"sim.mu2e.Beam.MDC2025ac.art": 5000}, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh" + }, + { + "desc": "BeamSplitter-000", + "dsconf": "Run1Bai", + "fcl": "Production/JobConfig/beam/BeamSplitter.fcl", + "input_data": {"sim.mu2e.Beam.Run1Bai.art": 100}, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + }, + { + "desc": "TargetStopsCat", + "dsconf": "Run1Baa1", + "input_data": {"sim.mu2e.TargetStops.Run1Baa1.art": 2500}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "sim.owner.TargetStopsCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh" + }, + { + "desc": "MuonStopSelector", + "dsconf": "Run1Bai", + "input_data": {"sim.mu2e.TargetStopsCat.Run1Bai-003.art": 1}, + "fcl": "Production/JobConfig/pileup/MuonStopSelector.fcl", + "inloc": "tape", + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + }, + { + "desc": "MuStopPileupCat", + "dsconf": "Run1Baa", + "input_data": {"dts.mu2e.MuStopPileup.Run1Baa.art": 100}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.MuStopPileupCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh" + }, + { + "desc": "MuStopPileupCat", + "dsconf": "Run1Bag", + "input_data": {"dts.mu2e.MuStopPileup.Run1Bag.art": 1000}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.MuStopPileupCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh" + }, + { + "desc": "NeutralsFlashCat", + "dsconf": "Run1Baa", + "input_data": {"dts.mu2e.NeutralsFlash.Run1Baa.art": 5000}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.NeutralsFlashCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh" + }, + { + "desc": "NeutralsFlashCat", + "dsconf": "Run1Bag", + "input_data": {"dts.mu2e.NeutralsFlash.Run1Bag.art": 5000}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.NeutralsFlashCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh" + }, + { + "desc": "NeutralsCat", + "dsconf": "Run1Baa", + "input_data": {"sim.mu2e.Neutrals.MDC2025ae3.art": 500}, + "fcl": "Production/JobConfig/common/artcat.fcl", + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.NeutralsCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh" + } +] diff --git a/data/Run1B/mix.json b/data/Run1B/mix.json new file mode 100644 index 0000000..8882f69 --- /dev/null +++ b/data/Run1B/mix.json @@ -0,0 +1,460 @@ +[ + { + "input_data": [ + {"dts.mu2e.CeEndpoint.Run1Baa.art": 1}, + {"dts.mu2e.NoPrimary.Run1Baa.art": 1}, + {"dts.mu2e.FlateMinus.Run1Baa.art": 1}, + {"dts.mu2e.DIOtail50.Run1Baa.art": 1}, + {"dts.mu2e.DIOtail80.Run1Baa.art": 1}, + {"dts.mu2e.DIOtail90.Run1Baa.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Baa1.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Baa.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Baa.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Baa.art": 2 + }], + "dsconf": ["Run1Bab_best_v1_2"], + "mixconf": [0], + "pbeam": ["MixLow"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bab/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["tape"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_2", + "physics.filters.NeutralsFlashMixer.mu2e.simStageEfficiencyTags": [], + "physics.filters.NeutralsFlashMixer.mu2e.meanEventsPerProton": 1.023e-05, + "outputs.TriggerableOutput.SelectEvents" : [], + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt" + } + ] + }, + { + "input_data": [ + {"dts.mu2e.CeEndpoint.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail0_60.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail60_80.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail80_90.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail90_inf.Run1Bab.art": 1}, + {"dts.mu2e.FlatGamma.Run1Bab.art": 1}, + {"dts.mu2e.FlateMinus.Run1Bab.art": 1}, + {"dts.mu2e.NoPrimary.Run1Bab.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Baa1.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Baa.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Baa.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Baa.art": 2 + }], + "dsconf": ["Run1Bab2_best_v1_2"], + "mixconf": [0], + "pbeam": ["MixLow"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bab/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["tape"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_2", + "physics.filters.NeutralsFlashMixer.mu2e.simStageEfficiencyTags": [], + "physics.filters.NeutralsFlashMixer.mu2e.meanEventsPerProton": 1.023e-05, + "outputs.TriggerableOutput.SelectEvents" : [], + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650" + } + ] + } + , + { + "input_data": [ + {"dts.mu2e.CeEndpoint.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail0_60.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail60_80.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail80_90.Run1Bab.art": 1}, + {"dts.mu2e.DIOtail90_inf.Run1Bab.art": 1}, + {"dts.mu2e.FlatGamma.Run1Bab.art": 1}, + {"dts.mu2e.FlateMinus.Run1Bab.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Baa1.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Baa.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Baa.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Baa.art": 2 + }], + "dsconf": ["Run1Baf_best_v1_4-000"], + "mixconf": [0], + "pbeam": ["MixLow"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baf/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["stash"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "physics.producers.makeSD.UseStatus": "true", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "#include": "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "physics.producers.PBISim.extendedMean": "1.58e6", + "physics.producers.PBISim.cutMax": "9.48e6" + } + ] + } + , + { + "input_data": [ + {"dts.mu2e.NoPrimary.Run1Bab.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Baa1.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Baa.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Baa.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Baa.art": 2 + }], + "dsconf": ["Run1Baf_best_v1_4-000"], + "mixconf": [0], + "pbeam": ["MixLow"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baf/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["stash"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "physics.producers.PBISim.extendedMean": "1.58e6", + "physics.producers.PBISim.cutMax": "9.48e6", + "physics.producers.makeSD.UseStatus": "true", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art" + } + ] + } + , + { + "input_data": [ + {"dts.mu2e.CeEndpoint.Run1Bag.art": 1}, + {"dts.mu2e.DIOtail0_60.Run1Bag.art": 1}, + {"dts.mu2e.DIOtail60_80.Run1Bag.art": 1}, + {"dts.mu2e.DIOtail80_90.Run1Bag.art": 1}, + {"dts.mu2e.DIOtail90_inf.Run1Bag.art": 1}, + {"dts.mu2e.FlatGamma.Run1Bag.art": 1}, + {"dts.mu2e.FlateMinus.Run1Bag.art": 1}, + {"dts.mu2e.MuCap1809keVCalo.Run1Bag.art": 1}, + {"dts.mu2e.RPCExternal.Run1Bah.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Bag.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Bag.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Bag.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Bag.art": 2 + }], + "dsconf": ["Run1Bah_best_v1_4-000"], + "mixconf": [0], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["resilient"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "#include": "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "physics.producers.PBISim.extendedMean": "5.92e6", + "physics.producers.PBISim.cutMax": "35.5e6", + "physics.filters.NeutralsFlashMixer.mu2e.simStageEfficiencyTags": [], + "physics.filters.NeutralsFlashMixer.mu2e.meanEventsPerProton": 1.0e-07 + } + ] + } + , + { + "input_data": [ + {"dts.mu2e.NoPrimary.Run1Bag.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Bag.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Bag.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Bag.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Bag.art": 2 + }], + "dsconf": ["Run1Bah_best_v1_4-000"], + "mixconf": [0], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["resilient"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "physics.producers.PBISim.extendedMean": "5.92e6", + "physics.producers.PBISim.cutMax": "35.5e6", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "#include": [ + "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "Production/JobConfig/mixing/NoPrimary.fcl" + ] + } + ] + } + , + { + "input_data": [ + {"dts.mu2e.NoPrimary.Run1Bag.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Bag.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Bag.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Bag.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Bag.art": 2 + }], + "dsconf": ["Run1Bah_best_v1_4-002"], + "mixconf": [0], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["resilient"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "physics.producers.PBISim.extendedMean": "5.92e6", + "physics.producers.PBISim.cutMax": "35.5e6", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "#include": [ + "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "Production/JobConfig/mixing/NoPrimary.fcl" + ] + } + ] + }, + { + "input_data": [ + {"dts.mu2e.NoPrimary.Run1Bai.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Bai-007.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Bai-007.art": 1, + "dts.mu2e.NeutralsFlashCat.Run1Bai-003.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Bai-007.art": 1 + }], + "dsconf": ["Run1Bai_best_v1_4-003"], + "mixconf": [0], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["tape"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.DbService.textFile": ["Production/data/SimEfficiencies2_Run1Bai.txt"], + "services.DbService.nearestMatch": "true", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "physics.producers.PBISim.extendedMean": "5.92e6", + "physics.producers.PBISim.cutMax": "35.5e6", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "outputs.Output.RejectEvents" : [], + "outputs.Output.SelectEvents" : [], + "#include": [ + "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "Production/JobConfig/mixing/NoPrimary.fcl" + ] + } + ] + } + , + { + "input_data": [ + {"dts.mu2e.CeEndpoint.Run1Bai.art": 1}, + {"dts.mu2e.FlateMinus.Run1Bai-001.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Bai-003.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Bai-003.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Bai-003.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Bai-004.art": 2 + }], + "dsconf": ["Run1Bai_best_v1_4-000"], + "mixconf": [0], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["resilient"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.DbService.textFile": ["Production/data/SimEfficiencies2_Run1Bai.txt"], + "services.DbService.nearestMatch": "true", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + + + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "outputs.Output.RejectEvents" : [], + "outputs.Output.SelectEvents" : [], + "#include": "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "physics.producers.PBISim.extendedMean": "5.92e6", + "physics.producers.PBISim.cutMax": "35.5e6" + } + ] + }, + { + "input_data": [ + {"dts.mu2e.RPCExternal.Run1Bai-001.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Bai-003.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Bai-003.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Bai-003.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Bai-004.art": 2 + }], + "dsconf": ["Run1Bai_best_v1_4-001"], + "mixconf": [0], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["resilient"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.DbService.textFile": ["Production/data/SimEfficiencies2_Run1Bai.txt"], + "services.DbService.nearestMatch": "true", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "services.ProditionsService.strawElectronics.digitizationStart": "300", + + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationStart": "300", + "physics.producers.CaloDigiMaker.digitizationStart": "300", + + + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "outputs.Output.RejectEvents" : [], + "outputs.Output.SelectEvents" : [], + "#include": "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "physics.producers.PBISim.extendedMean": "5.92e6", + "physics.producers.PBISim.cutMax": "35.5e6" + } + ] + }, + { + "input_data": [ + {"dts.mu2e.NoPrimary.Run1Bai.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.Run1Bai-003.art": 1, + "dts.mu2e.EleBeamFlashCat.Run1Bai-003.art": 25, + "dts.mu2e.NeutralsFlashCat.Run1Bai-003.art": 1, + "dts.mu2e.MuStopPileupCat.Run1Bai-004.art": 2 + }], + "dsconf": ["Run1Bai_best_v1_4-002"], + "mixconf": [0], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["tape"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.DbService.textFile": ["Production/data/SimEfficiencies2_Run1Bai.txt"], + "services.DbService.nearestMatch": "true", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "services.ProditionsService.strawElectronics.digitizationEnd": "1650", + "physics.producers.CaloShowerROMaker.digitizationEnd": "1650", + "physics.producers.CaloDigiMaker.digitizationEnd": "1650", + + "services.ProditionsService.strawElectronics.digitizationStart": "300", + "physics.producers.CaloShowerROMaker.digitizationStart": "300", + "physics.producers.CaloDigiMaker.digitizationStart": "300", + + "physics.producers.PBISim.extendedMean": "5.92e6", + "physics.producers.PBISim.cutMax": "35.5e6", + "outputs.Output.fileName": "dig.owner.{desc}.version.sequencer.art", + "outputs.Output.RejectEvents" : [], + "outputs.Output.SelectEvents" : [], + "#include": [ + "Production/JobConfig/digitize/OnSpill_run1b_epilog.fcl", + "Production/JobConfig/mixing/NoPrimary.fcl" + ] + } + ] + } +] \ No newline at end of file diff --git a/data/Run1B/primary_muon.json b/data/Run1B/primary_muon.json new file mode 100644 index 0000000..1637245 --- /dev/null +++ b/data/Run1B/primary_muon.json @@ -0,0 +1,287 @@ +[ + { + "desc": "DIO", + "dsconf": "Run1Baa", + "fcl": "Production/JobConfig/primary/DIOtail.fcl", + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Baa.art": 1}, + "events": 5000, + "njobs": 200, + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "physics.producers.generate.decayProducts.spectrum.elow": 0, + "physics.producers.generate.decayProducts.spectrum.ehi": 1000, + "outputs.PrimaryOutput.fileName": "dts.owner.DIO.version.sequencer.art", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 0.01, + "physics.filters.PrimaryFilter.MinimumPartMom": 1.0 + }, + "run": 1440, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh", + "owner": "mu2e", + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "DIOtail0_60", + "dsconf": "Run1Bag", + "fcl": "Production/JobConfig/primary/DIOtail.fcl", + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Baa.art": 1}, + "events": 1000000, + "njobs": 15000, + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "physics.producers.generate.decayProducts.spectrum.elow": 0, + "physics.producers.generate.decayProducts.spectrum.ehi": 60, + "physics.producers.generate.decayProducts.czmin": 0.9, + "physics.producers.generate.decayProducts.czmax": 1.0, + "outputs.PrimaryOutput.fileName": "dts.owner.DIOtail0_60.version.sequencer.art", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 20, + "physics.filters.PrimaryFilter.MinimumPartMom": 20 + }, + "run": 1451, + "version": 1, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "DIOtail60_80", + "dsconf": "Run1Bag", + "fcl": "Production/JobConfig/primary/DIOtail.fcl", + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Baa.art": 1}, + "events": 200000, + "njobs": 10000, + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "physics.producers.generate.decayProducts.spectrum.elow": 60, + "physics.producers.generate.decayProducts.spectrum.ehi": 80, + "physics.producers.generate.decayProducts.czmin": 0.9, + "physics.producers.generate.decayProducts.czmax": 1.0, + "outputs.PrimaryOutput.fileName": "dts.owner.DIOtail60_80.version.sequencer.art", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 20, + "physics.filters.PrimaryFilter.MinimumPartMom": 20 + }, + "run": 1451, + "version": 1, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "DIOtail80_90", + "dsconf": "Run1Bag", + "fcl": "Production/JobConfig/primary/DIOtail.fcl", + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Baa.art": 1}, + "events": 200000, + "njobs": 5000, + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "physics.producers.generate.decayProducts.spectrum.elow": 80, + "physics.producers.generate.decayProducts.spectrum.ehi": 90, + "physics.producers.generate.decayProducts.czmin": 0.9, + "physics.producers.generate.decayProducts.czmax": 1.0, + "outputs.PrimaryOutput.fileName": "dts.owner.DIOtail80_90.version.sequencer.art", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 20, + "physics.filters.PrimaryFilter.MinimumPartMom": 20 + }, + "run": 1450, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "DIOtail90_inf", + "dsconf": "Run1Bag", + "fcl": "Production/JobConfig/primary/DIOtail.fcl", + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Baa.art": 1}, + "events": 200000, + "njobs": 5000, + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "physics.producers.generate.decayProducts.spectrum.elow": 90, + "physics.producers.generate.decayProducts.spectrum.ehi": 1000, + "physics.producers.generate.decayProducts.czmin": 0.9, + "physics.producers.generate.decayProducts.czmax": 1.0, + "outputs.PrimaryOutput.fileName": "dts.owner.DIOtail90_inf.version.sequencer.art", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 20, + "physics.filters.PrimaryFilter.MinimumPartMom": 20 + }, + "run": 1450, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "CeEndpoint", + "dsconf": "Run1Bai-001", + "fcl": "Production/JobConfig/primary/CeEndpoint.fcl", + "fcl_overrides": { + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 20, + "physics.filters.PrimaryFilter.MinimumPartMom": 0 + }, + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Bai.art": 1}, + "events": 1000000, + "njobs": 2000, + "run": 1460, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "FlateMinus", + "dsconf": "Run1Bai-001", + "fcl": "Production/JobConfig/primary/FlateMinus.fcl", + "fcl_overrides": { + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "physics.producers.generate.startMom": 50, + "physics.producers.generate.endMom": 110, + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 20, + "physics.filters.PrimaryFilter.MinimumPartMom": 0 + }, + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Bai.art": 1}, + "njobs": 2000, + "events": 1000000, + "run": 1460, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "FlatGamma", + "dsconf": "Run1Bai-001", + "fcl": "Production/JobConfig/primary/FlatGamma.fcl", + "fcl_overrides": { + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.producers.generate.startMom": 50, + "physics.producers.generate.endMom": 110, + "physics.filters.PrimaryFilter.MinimumSumCaloStepE" : 20, + "physics.filters.PrimaryFilter.MinimumPartMom": 0 + }, + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Bai.art": 1}, + "njobs": 2000, + "events": 1000000, + "run": 1460, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "NoPrimary", + "dsconf": "Run1Bai", + "fcl": "Production/JobConfig/primary/NoPrimary.fcl", + "fcl_overrides": { + "physics.producers.compressDetStepMCs.surfaceStepTag" : "FindMCPrimary", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt" + }, + "events": 5000, + "njobs": 20000, + "run": 1460, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh", + "owner": "mu2e", + "inloc": "disk", + "outloc": { + "*.art": "disk" + } + }, + { + "desc": "MuCap1809keV", + "dsconf": "Run1Bag", + "fcl": "Production/JobConfig/primary/MuCap1809keVCalo.fcl", + "fcl_overrides": { + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt" + }, + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Baa.art": 1}, + "events": 10000, + "njobs": 1000, + "run": 1450, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bag/setup.sh", + "owner": "mu2e", + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "CosmicCRYAll", + "dsconf": "Run1Bah", + "fcl": "Production/JobConfig/cosmic/S2Resampler.fcl", + "fcl_overrides": { + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "outputs.PrimaryOutput.fileName": "dts.owner.CosmicCRYAll.version.sequencer.art" + }, + "resampler_name": "CosmicResampler", + "input_data": {"sim.mu2e.CosmicDSStopsCRYAll.MDC2025ab.art": 1}, + "events": 500000, + "njobs": 10000, + "run": 1450, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh", + "owner": "mu2e", + "inloc": "tape", + "outloc": { + "*.art": "tape" + }, + "sequential_aux": true + } +] diff --git a/data/Run1B/reco.json b/data/Run1B/reco.json new file mode 100644 index 0000000..8f88f1f --- /dev/null +++ b/data/Run1B/reco.json @@ -0,0 +1,169 @@ +[ + { + "dsconf": "Run1Bab2_best_v1_2", + "fcl": "Production/JobConfig/recoMC/NoField.fcl", + "input_data": [ + {"dig.mu2e.CeEndpointOnSpillTriggerable.Run1Bab_best_v1_2.art": 100}, + {"dig.mu2e.CeEndpointMixLowTriggerable.Run1Bab_best_v1_2.art": 100}, + + {"dig.mu2e.DIOOnSpillTriggerable.Run1Bab_best_v1_2.art": 100}, + + {"dig.mu2e.DIOtail50OnSpillTriggerable.Run1Bab_best_v1_2.art": 100}, + {"dig.mu2e.DIOtail50MixLowTriggerable.Run1Bab_best_v1_2.art": 100}, + + {"dig.mu2e.DIOtail80OnSpillTriggerable.Run1Bab_best_v1_2.art": 100}, + {"dig.mu2e.DIOtail80MixLowTriggerable.Run1Bab_best_v1_2.art": 100}, + + {"dig.mu2e.DIOtail90OnSpillTriggerable.Run1Bab_best_v1_2.art": 100}, + {"dig.mu2e.DIOtail90MixLowTriggerable.Run1Bab_best_v1_2.art": 100}, + + {"dig.mu2e.FlateMinusMixLowTriggerable.Run1Bab_best_v1_2.art": 100}, + {"dig.mu2e.NoPrimaryMixLowTriggerable.Run1Bab_best_v1_2.art": 100} + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_2", + "outputs.KinematicLineOutput.fileName": "mcs.mu2e.{desc}-KL.version.sequencer.art", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt" + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bab/setup.sh" + }, + + { + "dsconf": "Run1Baf_best_v1_4-000", + "fcl": "Production/JobConfig/recoMC/NoFieldRun1B.fcl", + "input_data": [ + {"dig.mu2e.CeEndpointMixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + {"dig.mu2e.DIOtail0_60MixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + {"dig.mu2e.DIOtail60_80MixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + {"dig.mu2e.DIOtail80_90MixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + {"dig.mu2e.DIOtail90_infMixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + {"dig.mu2e.FlatGammaMixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + {"dig.mu2e.FlateMinusMixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + {"dig.mu2e.NoPrimaryMixLowTriggerable.Run1Bab2_best_v1_2.art": 10}, + + {"dig.mu2e.CeEndpointMixLow.Run1Baf_best_v1_4.art": 10}, + {"dig.mu2e.DIOtail0_60MixLow.Run1Baf_best_v1_4.art": 10}, + {"dig.mu2e.DIOtail60_80MixLow.Run1Baf_best_v1_4.art": 10}, + {"dig.mu2e.DIOtail80_90MixLow.Run1Baf_best_v1_4.art": 10}, + {"dig.mu2e.DIOtail90_infMixLow.Run1Baf_best_v1_4.art": 10}, + {"dig.mu2e.FlatGammaMixLow.Run1Baf_best_v1_4.art": 10}, + {"dig.mu2e.FlateMinusMixLow.Run1Baf_best_v1_4.art": 10}, + {"dig.mu2e.NoPrimaryMixLow.Run1Baf_best_v1_4.art": 10}, + + {"dig.mu2e.CeEndpointOnSpillTriggerable.Run1Bab2_best_v1_2.art": 1}, + {"dig.mu2e.DIOtail0_60OnSpillTriggerable.Run1Bab2_best_v1_2.art": 1}, + {"dig.mu2e.DIOtail60_80OnSpillTriggerable.Run1Bab2_best_v1_2.art": 1}, + {"dig.mu2e.DIOtail80_90OnSpillTriggerable.Run1Bab2_best_v1_2.art": 1}, + {"dig.mu2e.DIOtail90_infOnSpillTriggerable.Run1Bab2_best_v1_2.art": 1}, + {"dig.mu2e.FlatGammaOnSpillTriggerable.Run1Bab2_best_v1_2.art": 1}, + {"dig.mu2e.FlateMinusOnSpillTriggerable.Run1Bab2_best_v1_2.art": 1} + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "outputs.KinematicLineOutput.fileName": "mcs.mu2e.{desc}-KL.version.sequencer.art", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt" + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baf/setup.sh" + }, + { + "dsconf": "Run1Baf_best_v1_4-001", + "fcl": "Production/JobConfig/recoMC/NoFieldRun1B.fcl", + "input_data": [ + {"dig.mu2e.CeEndpointMixLow.Run1Baf_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail0_60MixLow.Run1Baf_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail60_80MixLow.Run1Baf_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail80_90MixLow.Run1Baf_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail90_infMixLow.Run1Baf_best_v1_4-000.art": 10}, + {"dig.mu2e.FlatGammaMixLow.Run1Baf_best_v1_4-000.art": 10}, + {"dig.mu2e.FlateMinusMixLow.Run1Baf_best_v1_4-000.art": 10}, + {"dig.mu2e.NoPrimaryMixLow.Run1Baf_best_v1_4-000.art": 10} + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "outputs.KinematicLineOutput.fileName": "mcs.mu2e.{desc}-KL.version.sequencer.art", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt" + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baf/setup.sh" + }, + { + "dsconf": "Run1Bah_best_v1_4-002", + "tarball_append": "-reco", + "fcl": "Production/JobConfig/recoMC/NoFieldRun1B.fcl", + "input_data": [ + {"dig.mu2e.CeEndpointMix1BB.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail0_60Mix1BB.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail60_80Mix1BB.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail80_90Mix1BB.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail90_infMix1BB.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.FlatGammaMix1BB.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.FlateMinusMix1BB.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.NoPrimaryMix1BB.Run1Bah_best_v1_4-002.art": 10}, + {"dig.mu2e.MuCap1809keVCaloMix1BB.Run1Bah_best_v1_4-000.art": 10}, + + {"dig.mu2e.CeEndpoint.Run1Bah_best_v1_4-000.art": 1}, + {"dig.mu2e.DIOtail0_60.Run1Bah_best_v1_4-000.art": 10}, + {"dig.mu2e.DIOtail60_80.Run1Bah_best_v1_4-000.art": 1}, + {"dig.mu2e.DIOtail80_90.Run1Bah_best_v1_4-000.art": 1}, + {"dig.mu2e.DIOtail90_inf.Run1Bah_best_v1_4-000.art": 1}, + {"dig.mu2e.FlatGamma.Run1Bah_best_v1_4-000.art": 1}, + {"dig.mu2e.FlateMinus.Run1Bah_best_v1_4-000.art": 1}, + {"dig.mu2e.NoPrimary.Run1Bah_best_v1_4-000.art": 1}, + {"dig.mu2e.MuCap1809keVCalo.Run1Bah_best_v1_4-000.art": 1} + + + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "outputs.KinematicLineOutput.fileName": "mcs.mu2e.{desc}-KL.version.sequencer.art", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt" + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh" + }, + { + "dsconf": "Run1Bai_best_v1_4-001", + "tarball_append": "-reco", + "fcl": "Production/JobConfig/recoMC/NoFieldRun1B.fcl", + "input_data": [ + {"dig.mu2e.CeEndpointMix1BB.Run1Bai_best_v1_4-000.art": 10}, + {"dig.mu2e.FlatGammaMix1BB.Run1Bai_best_v1_4-000.art": 10}, + {"dig.mu2e.FlateMinusMix1BB.Run1Bai_best_v1_4-000.art": 10}, + {"dig.mu2e.NoPrimaryMix1BB.Run1Bai_best_v1_4-000.art": 10}, + {"dig.mu2e.MuCap1809keVCaloMix1BB.Run1Bai_best_v1_4-000.art": 10}, + + {"dig.mu2e.CeEndpoint.Run1Bai_best_v1_4-000.art": 1}, + {"dig.mu2e.FlatGamma.Run1Bai_best_v1_4-000.art": 1}, + {"dig.mu2e.FlateMinus.Run1Bai_best_v1_4-000.art": 1}, + {"dig.mu2e.NoPrimary.Run1Bai_best_v1_4-000.art": 1}, + {"dig.mu2e.MuCap1809keVCalo.Run1Bai_best_v1_4-000.art": 1} + + + ], + "fcl_overrides": { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_4", + "services.DbService.nearestMatch": "true", + "outputs.KinematicLineOutput.fileName": "mcs.mu2e.{desc}-KL.version.sequencer.art", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt" + }, + "inloc": "tape", + "outloc": {"*.art": "tape"}, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh" + } +] diff --git a/data/Run1B/resampler_beam.json b/data/Run1B/resampler_beam.json new file mode 100644 index 0000000..3500f4a --- /dev/null +++ b/data/Run1B/resampler_beam.json @@ -0,0 +1,222 @@ +[ + { + "desc": "NeutralsFlash", + "dsconf": "Run1Baa", + "fcl": "Production/JobConfig/pileup/NeutralsResampler.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt" + }, + "resampler_name": "neutralsResampler", + "input_data": {"sim.mu2e.Neutrals.MDC2025ae3.art": 1}, + "njobs": 5000, + "events": 100000, + "run": 1440, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh", + "owner": "mu2e", + "inloc": "tape", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "MuBeamFlash", + "dsconf": "Run1Baa1", + "fcl": "Production/JobConfig/pileup/MuBeamResampler.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt" + }, + "resampler_name": "beamResampler", + "input_data": {"sim.mu2e.MuBeamCat.Run1Baa.art": 1}, + "njobs": 5000, + "events": 400000, + "run": 1440, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh", + "owner": "mu2e", + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "EleBeamFlash", + "dsconf": "Run1Baa", + "fcl": "Production/JobConfig/pileup/EleBeamResampler.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt" + }, + "resampler_name": "beamResampler", + "input_data": {"sim.mu2e.EleBeamCat.Run1Baa.art": 1}, + "njobs": 5000, + "events": 1000000, + "run": 1440, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh", + "owner": "mu2e", + "inloc": "disk", + "outloc": { + "*.art": "tape" + }, + "sequential_aux": true + }, + { + "desc": "MuStopPileup", + "dsconf": "Run1Baa", + "fcl": "Production/JobConfig/pileup/MuStopPileup.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v01.txt", + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt" + }, + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.MuminusStopsCat.Run1Baa.art": 1}, + "njobs": 500, + "events": 400000, + "run": 1440, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Baa/setup.sh", + "owner": "mu2e", + "inloc": "tape", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "PiTargetStops", + "dsconf": "Run1Bai", + "fcl": "Production/JobConfig/pileup/PiBeamResampler.fcl", + "resampler_name": "beamResampler", + "input_data": {"sim.mu2e.PiBeamCat.Run1Bah.art": 1}, + "inloc": "tape", + "outloc": { + "*.art": "tape" + }, + "njobs": 500, + "events": 200000, + "run": 1460, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh", + "owner": "mu2e", + "sequential_aux": true + }, + { + "desc": "PiMinusFilter", + "dsconf": "MDC2025ac", + "fcl_overrides": { + "physics.filters.PionFilter.simParticles": "TargetPiStopFilter", + "services.TFileService.fileName": "/dev/null" + }, + "fcl": "Production/JobConfig/primary/TargetPiStopPreFilter.fcl", + "input_data": {"sim.mu2e.PiTargetStops.MDC2025ac.art": 10}, + "inloc": "tape", + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh" + }, + { + "desc": "RPCInternalPhysical", + "dsconf": "MDC2025af", + "fcl": "Production/JobConfig/primary/RPCInternalPhysical.fcl", + "input_data": {"sim.mu2e.PhysicalPionStops.MDC2025ac.art": 1}, + "resampler_name": "TargetPiStopResampler", + "njobs": 5000, + "events": 25000, + "run": 1430, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025af/setup.sh", + "owner": "mu2e", + "sequential_aux": true + }, + { + "desc": "RPCExternalPhysical", + "dsconf": "Run1Bah", + "fcl": "Production/JobConfig/primary/RPCExternalPhysical.fcl", + "resampler_name": "TargetPiStopResampler", + "input_data": {"sim.mu2e.PhysicalPionStops.Run1Bah.art": 1}, + "njobs": 5000, + "events": 1000000, + "run": 1450, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh", + "owner": "mu2e", + "sequential_aux": true + }, + { + "desc": "RPCExternal", + "dsconf": "Run1Bai-001", + "fcl": "Production/JobConfig/primary/RPCExternal.fcl", + "fcl_overrides": { + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt", + "physics.filters.PrimaryFilter.StrawGasSteps" : [], + "physics.filters.PrimaryFilter.MinimumPartMom": 0, + "physics.filters.StopSelector.cuts.tmin": 0 + }, + "resampler_name": "TargetPiStopResampler", + "input_data": {"sim.mu2e.PiTargetStops.Run1Bai.art": 1}, + "njobs": 5000, + "events": 1000000, + "run": 1460, + "inloc": "disk", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh", + "owner": "mu2e", + "sequential_aux": true + }, + + { + "desc": "PhysicalPionStops", + "dsconf": "Run1Bah", + "fcl": "Production/JobConfig/beam/PhysicalPionStops.fcl", + "fcl_overrides": { + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "outputs.Output.fastCloning": false + }, + "resampler_name": "StopResampler", + "input_data": {"sim.mu2e.PiTargetStops.Run1Bah.art": 1}, + "njobs": 2000, + "events": 5000000, + "run": 1450, + "inloc": "tape", + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh", + "owner": "mu2e", + "sequential_aux": true + }, + { + "desc": "NeutralsFlashCat", + "dsconf": "MDC2025ad", + "sequencer_from_index": true, + "fcl": "Production/JobConfig/common/artcat.fcl", + "input_data": { + "dts.mu2e.NeutralsFlash.MDC2025ac.art": { + "count": 5000, + "random": true + } + }, + "njobs": 1000, + "inloc": "disk", + "fcl_overrides": { + "outputs.out.fileName": "dts.owner.NeutralsFlashCat.version.sequence.art" + }, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ad/setup.sh", + "owner": "mu2e" + } + +] \ No newline at end of file diff --git a/data/Run1B/stage1.json b/data/Run1B/stage1.json new file mode 100644 index 0000000..168d265 --- /dev/null +++ b/data/Run1B/stage1.json @@ -0,0 +1,99 @@ +[ + { + "desc": "POT_Run1_a", + "dsconf": "MDC2025ac", + "fcl": "Production/JobConfig/beam/POT.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_a.txt" + }, + "njobs": 20000, + "events": 5000, + "run": 1430, + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ac/setup.sh", + "owner": "mu2e" + }, + { + "desc": "POT_Run1_b", + "dsconf": "MDC2025ae", + "fcl": "Production/JobConfig/beam/POT.fcl", + "fcl_overrides": { + "services.GeometryService.bFieldFile": "Offline/Mu2eG4/geom/bfgeom_DSOff.txt" + }, + "njobs": 10000, + "events": 5000, + "run": 1430, + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ae/setup.sh", + "owner": "mu2e" + }, + { + "desc": "POT_Run1_b", + "dsconf": "Run1Bai", + "fcl": "Production/JobConfig/beam/POT.fcl", + "fcl_overrides": { + "#include": "Production/JobConfig/beam/epilog_1b.fcl", + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v06.txt" + }, + "njobs": 20000, + "events": 100000, + "run": 1460, + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bai/setup.sh", + "owner": "mu2e" + }, + { + "desc": "CosmicDSStopsCRY", + "dsconf": "MDC2025ab", + "fcl": "Production/JobConfig/cosmic/S1DSStopsCRY.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_a.txt", + "outputs.S1LowOutput.fileName": "/dev/null" + }, + "njobs": 20000, + "events": 2500000, + "run": 1430, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ab/setup.sh", + "owner": "mu2e" + }, + { + "desc": "ExtractedCRY", + "dsconf": "MDC2025ad", + "fcl": "Production/JobConfig/cosmic/ExtractedCRY.fcl", + "njobs": 20000, + "events": 500000, + "run": 1400, + "outloc": { + "*.art": "tape" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ad/setup.sh", + "owner": "mu2e" + }, + { + "desc": "PiBeam", + "dsconf": "Run1Bah", + "fcl": "Production/JobConfig/beam/POT_infinitepion.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_b_v03.txt", + "physics.producers.g4run.Mu2eG4CommonCut.pars[1].pars[2].pars": "[ TS2Vacuum, Coll31, Coll32, TS2InnerCryoShell, TS3InnerCryoShell, TS2CryoInsVac, TS3CryoInsVac, PbarAbsDisk, PbarAbsWedge, VirtualDetector_Coll31_In, VirtualDetector_Coll32_In, VirtualDetector_Coll31_Out, VirtualDetector_Coll32_Out, Coll31OutRecord, Coll32InRecord, Coll31OutRecord ]", + "physics.producers.g4run.Mu2eG4CommonCut.pars[1].pars[3].pars[0].pars[0].pars": "[ TS3Vacuum ]" + }, + "njobs": 5000, + "events": 200000, + "run": 1450, + "outloc": { + "*.art": "disk" + }, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/Run1Bah/setup.sh", + "owner": "mu2e" + } +] diff --git a/data/mdc2025/g4bl.json b/data/mdc2025/g4bl.json new file mode 100644 index 0000000..73d18a1 --- /dev/null +++ b/data/mdc2025/g4bl.json @@ -0,0 +1,14 @@ +[ + { + "runner": "g4bl", + "desc": "G4blPOT", + "dsconf": "TESTaa", + "embed_dir": "/exp/mu2e/app/users/oksuzian/G4BeamlineScripts", + "main_input": "Mu2E.in", + "events_per_job": 10000, + "njobs": 100, + "outputs": [ + {"dataset": "nts.mu2e.G4blPOT.TESTaa.root", "location": "disk"} + ] + } +] diff --git a/data/mdc2025/mix.json b/data/mdc2025/mix.json index 42ead37..f070280 100644 --- a/data/mdc2025/mix.json +++ b/data/mdc2025/mix.json @@ -86,5 +86,34 @@ } ] + }, + { + "input_data": [ + {"dts.mu2e.PBINormal_33344.MDC2025aj.art": 1}, + {"dts.mu2e.PBIPathological_33344.MDC2025aj.art": 1} + ], + "pileup_datasets": [{ + "dts.mu2e.MuBeamFlashCat.MDC2025ac.art": 1, + "dts.mu2e.EleBeamFlashCat.MDC2025ac.art": 25, + "dts.mu2e.NeutralsFlashCat.MDC2025ad.art": 1, + "dts.mu2e.MuStopPileupCat.MDC2025ac.art": 2 + }], + "dsconf": ["MDC2025ai_best_v1_3"], + "mixconf": [3], + "pbeam": ["Mix1BB"], + "owner": ["mu2e"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "merge_events": [500], + "inloc": ["resilient"], + "outloc": [{"dig.mu2e.*.art": "tape"}], + "fcl_overrides": [ + { + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_3", + "#include": "Production/JobConfig/mixing/NoPrimaryPBISequence.fcl", + "outputs.Output.fileName": "dig.mu2e.{desc}.{dsconf}.sequence.art" + } + ] } ] \ No newline at end of file diff --git a/data/mdc2025/pbi_sequence.json b/data/mdc2025/pbi_sequence.json index eeb5fa5..6604e93 100644 --- a/data/mdc2025/pbi_sequence.json +++ b/data/mdc2025/pbi_sequence.json @@ -1,7 +1,7 @@ [ { "desc": "PBINormal_33344", - "dsconf": "MDC2025ai", + "dsconf": "MDC2025aj", "fcl": "Production/JobConfig/primary/NoPrimaryPBISequence.fcl", "input_data": { "/cvmfs/mu2e.opensciencegrid.org/DataFiles/PBI/PBI_Normal_33344.txt": { @@ -12,7 +12,11 @@ "owner": "mu2e", "inloc": "none", "outloc": {"*.art": "tape"}, - "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh", + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025aj/setup.sh", + "event_id_per_index": { + "source.firstSubRunNumber": {"offset": 0, "step": 1}, + "source.firstEventNumber": {"offset": 0, "step": 1000} + }, "fcl_overrides": { "physics.producers.compressDetStepMCs.surfaceStepTag": "FindMCPrimary", "outputs.PrimaryOutput.fileName": "dts.owner.PBINormal_33344.version.sequencer.art" @@ -20,7 +24,7 @@ }, { "desc": "PBIPathological_33344", - "dsconf": "MDC2025ai", + "dsconf": "MDC2025aj", "fcl": "Production/JobConfig/primary/NoPrimaryPBISequence.fcl", "input_data": { "/cvmfs/mu2e.opensciencegrid.org/DataFiles/PBI/PBI_Pathological_33344.txt": { @@ -31,7 +35,11 @@ "owner": "mu2e", "inloc": "none", "outloc": {"*.art": "tape"}, - "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh", + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025aj/setup.sh", + "event_id_per_index": { + "source.firstSubRunNumber": {"offset": 0, "step": 1}, + "source.firstEventNumber": {"offset": 0, "step": 1000} + }, "fcl_overrides": { "physics.producers.compressDetStepMCs.surfaceStepTag": "FindMCPrimary", "outputs.PrimaryOutput.fileName": "dts.owner.PBIPathological_33344.version.sequencer.art" diff --git a/data/mdc2025/reco.json b/data/mdc2025/reco.json index 33c7be9..1843d8d 100644 --- a/data/mdc2025/reco.json +++ b/data/mdc2025/reco.json @@ -89,5 +89,22 @@ "inloc": "tape", "outloc": {"*.art": "disk"}, "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025af/setup.sh" + }, + { + "dsconf": ["MDC2025ai_best_v1_3"], + "tarball_append": "-reco", + "fcl": ["Production/JobConfig/recoMC/OnSpill.fcl"], + "input_data": [ + {"dig.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.art": 1}, + {"dig.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.art": 1} + ], + "fcl_overrides": [{ + "outputs.LoopHelixOutput.fileName": "mcs.owner.{desc}.version.sequencer.art", + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_3" + }], + "inloc": ["tape"], + "outloc": [{"*.art": "disk"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh"] } ] \ No newline at end of file diff --git a/data/mdc2025/resampler_stm.json b/data/mdc2025/resampler_stm.json new file mode 100644 index 0000000..6a1fe58 --- /dev/null +++ b/data/mdc2025/resampler_stm.json @@ -0,0 +1,90 @@ +[ + { + "desc": "STMBeamToVDEle", + "dsconf": "MDC2025ai", + "fcl": "Production/JobConfig/pileup/STM/BeamTo2VD.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_a.txt", + "outputs.compressedOutput101.fileName": "dts.mu2e.{desc}101.MDC2025ai.sequence.art", + "outputs.compressedOutput116.fileName": "dts.mu2e.{desc}116.MDC2025ai.sequence.art" + }, + "resampler_name": "beamResampler", + "input_data": {"sim.mu2e.EleBeamCat.MDC2025ab.art": 1}, + "njobs": 5000, + "events": 1106516, + "run": 1430, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh", + "owner": "mu2e", + "inloc": "tape", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "STMBeamToVDMu", + "dsconf": "MDC2025ai", + "fcl": "Production/JobConfig/pileup/STM/BeamTo2VD.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_a.txt", + "outputs.compressedOutput101.fileName": "dts.mu2e.{desc}101.MDC2025ai.sequence.art", + "outputs.compressedOutput116.fileName": "dts.mu2e.{desc}116.MDC2025ai.sequence.art" + }, + "resampler_name": "beamResampler", + "input_data": {"sim.mu2e.MuBeamCat.MDC2025ab.art": 1}, + "njobs": 20000, + "events": 200000, + "run": 1430, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "STMBeamToVDTarget", + "dsconf": "MDC2025ai", + "fcl": "Production/JobConfig/pileup/STM/BeamTo2VD1809.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_a.txt", + "outputs.compressedOutput101.fileName": "dts.mu2e.{desc}101.MDC2025ai.sequence.art", + "outputs.compressedOutput116.fileName": "dts.mu2e.{desc}116.MDC2025ai.sequence.art" + }, + "resampler_name": "TargetStopResampler", + "input_data": {"sim.mu2e.TargetStopsCat.MDC2025ac.art": 1}, + "njobs": 1000, + "events": 1000000, + "run": 1430, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + }, + { + "desc": "STMNeutralsToVD", + "dsconf": "MDC2025ai", + "fcl": "Production/JobConfig/pileup/STM/NeutralsTo2VD.fcl", + "fcl_overrides": { + "services.GeometryService.inputFile": "Offline/Mu2eG4/geom/geom_run1_a.txt", + "outputs.compressedOutput101.fileName": "dts.mu2e.{desc}101.MDC2025ai.sequence.art", + "outputs.compressedOutput116.fileName": "dts.mu2e.{desc}116.MDC2025ai.sequence.art" + }, + "resampler_name": "neutralsResampler", + "input_data": {"sim.mu2e.NeutralsCat.MDC2025ab.art": 1}, + "njobs": 20000, + "events": 100000, + "run": 1430, + "simjob_setup": "/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh", + "owner": "mu2e", + "inloc": "resilient", + "outloc": { + "*.art": "disk" + }, + "sequential_aux": true + } +] \ No newline at end of file diff --git a/test/g4bl_smoke_jobdesc.json b/test/g4bl_smoke_jobdesc.json new file mode 100644 index 0000000..5103827 --- /dev/null +++ b/test/g4bl_smoke_jobdesc.json @@ -0,0 +1,13 @@ +[ + { + "runner": "g4bl", + "desc": "G4blPOT", + "dsconf": "TESTaa", + "embed_dir": "/exp/mu2e/app/users/oksuzian/G4BeamlineScripts", + "main_input": "Mu2E.in", + "events_per_job": 10, + "outputs": [ + {"dataset": "nts.mu2e.G4blPOT.TESTaa.*.root", "location": "disk"} + ] + } +] diff --git a/test/g4bl_tarball_smoke_jobdesc.json b/test/g4bl_tarball_smoke_jobdesc.json new file mode 100644 index 0000000..a53b995 --- /dev/null +++ b/test/g4bl_tarball_smoke_jobdesc.json @@ -0,0 +1,11 @@ +[ + { + "tarball": "/tmp/cnf.mu2e.G4blPOT.TESTaa.0.tar", + "njobs": 1, + "runner": "g4bl", + "inloc": "none", + "outputs": [ + {"dataset": "nts.mu2e.G4blPOT.TESTaa.*.root", "location": "disk"} + ] + } +] diff --git a/utils/datasetFileList.py b/utils/datasetFileList.py index 3d620c4..601b12c 100644 --- a/utils/datasetFileList.py +++ b/utils/datasetFileList.py @@ -12,26 +12,11 @@ # Handle both module and standalone imports try: - from .job_common import get_samweb_wrapper + from .job_common import get_samweb_wrapper, Mu2eFilename except ImportError: # When running as standalone script sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) - from utils.job_common import get_samweb_wrapper - -class Mu2eFilename: - """Parse Mu2e filenames to get relative path components.""" - - def __init__(self, filename: str): - self.filename = filename - - def relpathname(self) -> str: - """Get relative pathname like the Perl Mu2eFilename->relpathname().""" - # Generate hash-based subdirectory using SHA256 (matches original Perl behavior) - import hashlib - hash_obj = hashlib.sha256(self.filename.encode()) - hash_hex = hash_obj.hexdigest() - hash_path = f"{hash_hex[:2]}/{hash_hex[2:4]}" - return f"{hash_path}/{self.filename}" + from utils.job_common import get_samweb_wrapper, Mu2eFilename class Mu2eDSName: """Parse Mu2e dataset names.""" diff --git a/utils/job_common.py b/utils/job_common.py index 6ac0b34..4324e76 100644 --- a/utils/job_common.py +++ b/utils/job_common.py @@ -43,6 +43,11 @@ def basename(self) -> str: """Return the basename of the file.""" return self.filename + def relpathname(self) -> str: + """SHA256 hash-prefixed relative path, matching Perl Mu2eFilename->relpathname().""" + h = hashlib.sha256(self.filename.encode()).hexdigest() + return f"{h[:2]}/{h[2:4]}/{self.filename}" + def remove_storage_prefix(path: str) -> str: """Remove storage system prefixes (enstore:, dcache:) from a file path. diff --git a/utils/jobdef.py b/utils/jobdef.py index ba57830..c06e79b 100644 --- a/utils/jobdef.py +++ b/utils/jobdef.py @@ -473,13 +473,16 @@ def parse_samplinginput(spec: str) -> Tuple[str, int, List[str]]: tbs['subrunkey'] = 'source.subRun' elif source_type == 'PBISequence': - # PBISequence: one text-chunk file per job. The module's pset validator - # accepts only fileNames + runNumber (plus static config like - # reconstitutedModuleLabel, integratedSummary, verbosity). Passing - # source.maxEvents / firstSubRunNumber / firstEventNumber is rejected. - # Sequencer uniqueness comes from the input chunk basename (e.g. the - # ".00" slot in dts.mu2e.PBINormal_33344.MDC2025ac.00.txt) — no - # subrunkey needed. + # PBISequence: one text-chunk file per job. Up to MDC2025ai the module's + # pset validator accepted only fileNames + runNumber (plus static config + # like reconstitutedModuleLabel, integratedSummary, verbosity) and + # rejected source.maxEvents / firstSubRunNumber / firstEventNumber. + # MDC2025aj (Offline PR #1799 + Production #533, merged 2026-04-15) adds + # firstSubRunNumber and firstEventNumber as optional atoms (default 0), + # so per-index offsets via `event_id_per_index` are now accepted there. + # source.maxEvents is still rejected. Sequencer uniqueness otherwise + # comes from the input chunk basename (e.g. the ".00" slot in + # dts.mu2e.PBINormal_33344.MDC2025ac.00.txt) — no subrunkey needed. has_inputs = bool(args_state.get('inputs_list')) has_chunk_mode = bool(config and config.get('chunk_mode')) if not (has_inputs or has_chunk_mode): diff --git a/utils/listNewDatasets.py b/utils/listNewDatasets.py index a4e799b..1db2dc0 100755 --- a/utils/listNewDatasets.py +++ b/utils/listNewDatasets.py @@ -3,10 +3,12 @@ import os import sys +import glob +import time import argparse from datetime import datetime, timedelta from collections import defaultdict -from typing import Dict, List, Optional +from typing import Dict, List, Optional, Tuple if __name__ == '__main__': sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) @@ -14,18 +16,69 @@ from samweb_wrapper import list_files +DEFAULT_POMS_DIR = "/exp/mu2e/app/users/mu2epro/production_manager/poms_map" + + +def _default_db_path() -> str: + repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) + return os.path.join(repo_root, "poms_data.db") + + +def _db_is_stale(db_path: str, poms_dir: str, lookback_days: int) -> Tuple[bool, str]: + """Return (stale, reason). DB is stale if any POMS map within the + lookback window has been modified since the DB file's mtime.""" + if not os.path.exists(db_path): + return True, "DB does not exist" + db_mtime = os.path.getmtime(db_path) + cutoff = time.time() - lookback_days * 86400 + newer = [] + for f in glob.glob(os.path.join(poms_dir, "MDC202*.json")): + m = os.path.getmtime(f) + if m > db_mtime and m > cutoff: + newer.append(os.path.basename(f)) + if newer: + sample = ", ".join(newer[:3]) + (f", +{len(newer) - 3} more" if len(newer) > 3 else "") + return True, f"{len(newer)} map(s) newer than DB ({sample})" + return False, "" + + +def _ensure_db_fresh(db_path: str, poms_dir: str, days: int, no_rebuild: bool) -> None: + """If the DB is stale, do an incremental rebuild covering the lookback + window. No-op when fresh. With no_rebuild, prints a warning instead.""" + stale, reason = _db_is_stale(db_path, poms_dir, days) + if not stale: + return + if no_rebuild: + print(f"WARNING: DB stale ({reason}); --no-rebuild specified, completeness may be inaccurate.") + return + print(f"DB stale: {reason}; running incremental rebuild...") + from db_builder import build_db + cutoff_dt = datetime.now() - timedelta(days=days) + t0 = time.time() + build_db("MDC202*", db_path, poms_dir=poms_dir, since=cutoff_dt) + print(f"Rebuild took {time.time() - t0:.1f}s.") + + class DatasetLister: """List and summarize recently created datasets from SAM.""" - def __init__(self, filetype: str = "art", days: int = 7, + def __init__(self, filetype: str = "art", days: int = 7, user: str = "mu2epro", show_size: bool = False, - custom_query: Optional[str] = None): + custom_query: Optional[str] = None, + completeness: bool = False, no_rebuild: bool = False, + db_path: Optional[str] = None, + poms_dir: str = DEFAULT_POMS_DIR): self.filetype = filetype self.days = days self.user = user self.show_size = show_size self.custom_query = custom_query self.ext = f".{filetype}" + self.completeness = completeness + self.no_rebuild = no_rebuild + self.db_path = db_path or _default_db_path() + self.poms_dir = poms_dir + self._db_session = None # opened lazily in run() if completeness enabled def build_query(self) -> str: if self.custom_query: @@ -70,18 +123,55 @@ def group_files_by_dataset(self, files: List[str]) -> Dict[str, int]: dataset = self.extract_dataset_name(filename) dataset_counts[dataset] += 1 return dict(dataset_counts) - + + def _get_completeness(self, dataset: str) -> str: + """Look up / for a dataset in the POMS DB. + Returns a short formatted string suitable for the table column.""" + if self._db_session is None: + return "-" + try: + from poms_db import Job, JobOutput, DatasetInfo + except ImportError: + return "-" + out = self._db_session.query(JobOutput).filter_by(dataset=dataset).first() + if not out: + return "—" # not produced via POMS + job = self._db_session.query(Job).filter_by(id=out.job_id).first() + info = self._db_session.query(DatasetInfo).filter_by(dataset_name=dataset).first() + if not job or job.njobs == 0 or info is None or info.nfiles is None: + return "?" + marker = "" if info.nfiles >= job.njobs else " INCOMPLETE" + return f"{info.nfiles}/{job.njobs}{marker}" + def run(self): + # Refresh the POMS DB before SAM queries if completeness is requested. + # Cheap when the DB is fresh; only does work when a map changed. + if self.completeness: + try: + import sqlalchemy # noqa: F401 + except ImportError: + print("WARNING: SQLAlchemy not found (run 'pyenv ana' after " + "'muse setup ops'); completeness column disabled.") + self.completeness = False + if self.completeness: + _ensure_db_fresh(self.db_path, self.poms_dir, self.days, self.no_rebuild) + try: + from poms_db import get_db_session + self._db_session = get_db_session(self.db_path) + except Exception as e: + print(f"WARNING: could not open POMS DB ({e}); completeness column disabled.") + self.completeness = False + query = self.build_query() files = list_files(query) - + if not files: print("No files found matching query.") return - + dataset_counts = self.group_files_by_dataset(files) sorted_datasets = sorted(dataset_counts.items()) - + # Print header print("------------------------------------------------") header = f"{'COUNT':>8} {'DATASET':<100}" @@ -89,9 +179,12 @@ def run(self): if self.show_size: header += f" {'FILE SIZE':>10}" divider += f" {'--------':>10}" + if self.completeness: + header += f" {'COMPLETENESS':<22}" + divider += f" {'------------':<22}" print(header) print(divider) - + # Print datasets for dataset, count in sorted_datasets: line = f"{count:>8} {dataset:<100}" @@ -99,8 +192,10 @@ def run(self): avg_size = self.get_average_filesize(dataset) size_str = f"{avg_size:>7} MB" if avg_size != "N/A" else f"{'N/A':>10}" line += f" {size_str}" + if self.completeness: + line += f" {self._get_completeness(dataset):<22}" print(line) - + print("------------------------------------------------") @@ -111,11 +206,20 @@ def main(): parser.add_argument('--user', default='mu2epro', help='Username filter (default: mu2epro)') parser.add_argument('--size', action='store_true', help='Show average file sizes') parser.add_argument('--query', help='Custom SAM query') + parser.add_argument('--completeness', action='store_true', + help='Append nfiles/expected column from POMS DB; auto-rebuilds DB if stale') + parser.add_argument('--no-rebuild', action='store_true', + help='With --completeness, skip auto-rebuild even if DB is stale (warn only)') + parser.add_argument('--db', default=None, help='POMS SQLite DB path (default: /poms_data.db)') + parser.add_argument('--poms-dir', default=DEFAULT_POMS_DIR, + help=f'POMS map directory (default: {DEFAULT_POMS_DIR})') args = parser.parse_args() - + lister = DatasetLister(filetype=args.filetype, days=args.days, user=args.user, - show_size=args.size, custom_query=args.query) - + show_size=args.size, custom_query=args.query, + completeness=args.completeness, no_rebuild=args.no_rebuild, + db_path=args.db, poms_dir=args.poms_dir) + lister.run() diff --git a/utils/prod_utils.py b/utils/prod_utils.py index d8a00cd..3bec203 100644 --- a/utils/prod_utils.py +++ b/utils/prod_utils.py @@ -4,6 +4,9 @@ import json import os import re +import shlex +import tarfile +import tempfile from pathlib import Path from datetime import datetime from .jobfcl import Mu2eJobFCL @@ -184,83 +187,6 @@ def write_fcl_template(base, overrides): # (strings get quotes, lists get proper syntax with double quotes) f.write(f'{key}: {json.dumps(val)}\n') -def parse_jobdef_fields(jobdefs_file, index=None): - """ - Extract job definition fields from a jobdefs file and index. - - Args: - jobdefs_file: Path to the jobdefs file - index: Index of the job definition to extract (optional, will extract from fname env var if not provided) - - Returns: - tuple: (tarfile, job_index, inloc, outloc) - """ - - #check token before proceeding - try: - run(f"httokendecode -H", shell=True) - except SystemExit: - print("Warning: Token validation failed. Please check your token.") - run("pwd", shell=True) - run("ls -ltr", shell=True) - - # Extract index from fname environment variable if not provided - if index is None: - fname = os.getenv("fname") - if not fname: - print("Error: fname environment variable is not set.") - sys.exit(1) - try: - index = int(fname.split('.')[4].lstrip('0') or '0') - except (IndexError, ValueError) as e: - print("Error: Unable to extract index from filename.") - sys.exit(1) - - if not os.path.exists(jobdefs_file): - print(f"Error: Jobdefs file {jobdefs_file} does not exist.") - sys.exit(1) - - jobdefs_list = make_jobdefs_list(Path(jobdefs_file)) - - if len(jobdefs_list) < index: - print(f"Error: Expected at least {index} job definitions, but got only {len(jobdefs_list)}") - sys.exit(1) - - # Get the index-th job definition (adjusting for Python's 0-index). - jobdef = jobdefs_list[index] - print(f"The {index}th job definition is: {jobdef}") - - # Split the job definition into fields (parfile job_index inloc outloc). - fields = jobdef.split() - if len(fields) != 4: - print(f"Error: Expected 4 fields (parfile job_index inloc outloc) in the job definition, but got: {jobdef}") - sys.exit(1) - - # Return the fields: (tarfile, job_index, inloc, outloc) - print(f"IND={fields[1]} TARF={fields[0]} INLOC={fields[2]} OUTLOC={fields[3]}") - return fields[0], int(fields[1]), fields[2], fields[3] - -def make_jobdefs_list(input_file): - """ - Create a list of individual job definitions from a jobdef jobdesc file. - - Args: - input_file: Path to jobdef jobdesc file - - Returns: - List of individual job definition strings: parfile job_index inloc outloc - """ - if not input_file.exists(): - sys.exit(f"Input file not found: {input_file}") - - jobdefs_list = [] - for line in input_file.read_text().splitlines(): - parfile, njobs, inloc, outloc = line.strip().split() - for i in range(int(njobs)): - jobdefs_list.append(f"{parfile} {i} {inloc} {outloc}") - print(f"Generated the list of {len(jobdefs_list)} jobdefs from {input_file}") - return jobdefs_list - def replace_file_extensions(input_str, first_field, last_field): """Replace the first and last fields in a dot-separated string.""" fields = input_str.split('.') @@ -307,6 +233,30 @@ def validate_jobdesc(jobdesc): print("Error: No job descriptions found in jobdesc file") sys.exit(1) + # Check if g4bl runner (has runner: 'g4bl' field) + if jobdesc[0].get('runner') == 'g4bl': + if len(jobdesc) > 1: + print("Error: g4bl runner requires exactly one entry in jobdesc list") + sys.exit(1) + entry = jobdesc[0] + # The map (jobdesc) only carries dispatch fields. The runtime config + # (desc/dsconf/main_input/events_per_job) lives inside the tarball's + # `jobpars.json` for grid mode — the tarball is self-describing. + # Embed_dir mode (local smoke) has no tarball, so the entry must + # carry the runtime config directly. + if entry.get('tarball'): + required_fields = ['outputs'] # runtime config in tarball/jobpars.json + elif entry.get('embed_dir'): + required_fields = ['desc', 'dsconf', 'main_input', 'events_per_job', 'outputs'] + else: + print("Error: g4bl runner requires either 'tarball' or 'embed_dir'") + sys.exit(1) + for field in required_fields: + if field not in entry: + print(f"Error: g4bl runner requires '{field}' field") + sys.exit(1) + return 'g4bl' + # Check if template mode (has fcl_template field) if 'fcl_template' in jobdesc[0]: if len(jobdesc) > 1: @@ -625,6 +575,148 @@ def process_jobdef(jobdesc, fname, args): outputs = jobdesc_entry['outputs'] return fcl, simjob_setup, infiles, outputs, inloc + +def build_mu2e_cmd(fcl, simjob_setup, args): + """Build the `subprocess.run(..., shell=False)`-ready arg list for running + mu2e against an FCL. + + The inner bash script joins setup-source and mu2e with `&&` so mu2e is + skipped if the source fails — matches the prior shell=True + `f"source X && mu2e -c Y"` semantics. shell=False here closes the + quoting hazard around `fcl` / `simjob_setup` paths without changing + bash's parsing of the inner script. + """ + inner = f"source {simjob_setup} && mu2e -c {fcl}" + if args.nevts > 0: + inner += f" -n {int(args.nevts)}" + if args.mu2e_options.strip(): + inner += f" {args.mu2e_options.strip()}" + return ['bash', '-c', inner] + + +def process_g4bl_jobdef(jobdesc_entry, fname, args): + """Run a G4Beamline simulation job. Returns + (outputs, histo_file, log_file, succeeded). + + Two source modes: + - `tarball`: extract the cnf.*.tar (built by g4bl_jobdef build tool) to a + scratch dir; treat extracted `work/` as embed_dir. This is the grid path + since /exp/mu2e/app is not mounted on workers. + - `embed_dir`: read the lattice files directly from local fs. Local-only + smoke path; prefer tarball mode for any real run. + + Streams g4bl stdout/stderr to a SAM-named log file + (`log.mu2e....log`) in addition to the runner's + stdout. The log file is always returned (and exists) if exec started, even + on g4bl failure — push it via push_logs(log_file=...) so failed jobs are + debuggable in SAM. Raises RuntimeError on prep failures (missing tarball, + missing embed_dir, missing main_input) — no log produced in those cases. + + Sequencer: First_Event = job_index * events_per_job + 1 (0-based). + """ + parts = Path(fname).name.split('.') + if len(parts) < 5: + raise RuntimeError(f"Invalid g4bl fname: {fname}; expected 6-field Mu2e name") + sequencer = parts[4] + stripped = sequencer.lstrip('0') + job_index = int(stripped) if stripped else 0 + + # Tarball mode (grid path): runtime config lives in jobpars.json INSIDE + # the tarball — the tarball is self-describing. Embed_dir mode (local + # smoke) has no tarball, so config is on the jobdesc entry. + tarball = jobdesc_entry.get('tarball') + if tarball: + if not Path(tarball).is_file(): + raise RuntimeError(f"tarball not found: {tarball}") + extract_dir = tempfile.mkdtemp(prefix='g4bl_extract_') + with tarfile.open(tarball) as t: + t.extractall(extract_dir) + embed_dir = os.path.join(extract_dir, 'work') + if not Path(embed_dir).is_dir(): + raise RuntimeError(f"tarball missing 'work/' subdir: {tarball}") + jobpars_path = os.path.join(extract_dir, 'jobpars.json') + if not Path(jobpars_path).is_file(): + raise RuntimeError(f"tarball missing jobpars.json: {tarball}") + with open(jobpars_path) as f: + jobpars = json.load(f) + # Required keys in jobpars.json — fail loudly if any is missing. + desc = jobpars['desc'] + dsconf = jobpars['dsconf'] + main_input = jobpars['main_input'] + events_per_job = int(jobpars['events_per_job']) + else: + embed_dir = jobdesc_entry['embed_dir'] + desc = jobdesc_entry['desc'] + dsconf = jobdesc_entry['dsconf'] + main_input = jobdesc_entry['main_input'] + events_per_job = int(jobdesc_entry['events_per_job']) + if not Path(embed_dir).is_dir(): + raise RuntimeError(f"embed_dir not found: {embed_dir}") + + if not (Path(embed_dir) / main_input).is_file(): + raise RuntimeError(f"main_input not found: {embed_dir}/{main_input}") + + first_event = job_index * events_per_job + 1 + + # SAM-named histogram + log files. `nts.` (simulation ntuple) is the + # canonical Mu2e tier for ROOT TTrees from a sim job; matches the + # metacat naming convention used everywhere else. + histo_file = f"nts.mu2e.{desc}.{dsconf}.{sequencer}.root" + histo_path = os.path.abspath(histo_file) + log_file = f"log.mu2e.{desc}.{dsconf}.{sequencer}.log" + log_path = os.path.abspath(log_file) + + # Native AL9 g4bl via spack. spack is a shell function defined by + # setupmu2e-art.sh; `spack load g4beamline` directly fails to propagate + # PATH in non-interactive shells, so use `eval $(spack load --sh ...)`. + # No apptainer wrap, no SL7 container — workers already run on AL9 + # (fnal-wn-el9) per the standard fermigrid.cfg outer container. + # `unset PYTHON*` avoids subprocess-leaked vars (PYTHONHOME/PATH from + # the runmu2e Python) confusing spack (which is itself Python and looks + # up packages via its own site-packages). Same class of leak we hit + # with apptainer; fixed there with --cleanenv, here by selective unset. + # CLI keyword syntax: plain `key=value`, NOT `param key=value` (the + # `param` form is input-file syntax; g4bl 3.08b rejects it on the + # command line, unlike the older 3.08 SL7 build that was lenient). + inner_script = ( + # Unset SPACK_ENV first: if the parent shell did `muse setup ops` + # (the typical art-runner setup), SPACK_ENV=...ops-019... is + # inherited via subprocess. `spack load g4beamline` then searches + # only the ops-019 environment — which doesn't contain g4beamline — + # and fails with "Spec 'g4beamline' matches no installed packages". + # The Mu2e wiki notes this as a known limitation ("after muse setup + # it is no longer possible to spack load a package"). Discovered + # 2026-04-28 after env-leak debugging. + "unset SPACK_ENV PYTHONHOME PYTHONPATH PYTHONNOUSERSITE\n" + "source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh > /dev/null 2>&1\n" + 'eval "$(spack load --sh g4beamline)"\n' + f"cd {shlex.quote(embed_dir)}\n" + f"g4bl {shlex.quote(main_input)} viewer=none " + f"First_Event={first_event} Num_Events={events_per_job} " + f"histoFile={shlex.quote(histo_path)}" + ) + cmd_list = ['bash', '-c', inner_script] + + print(f"g4bl: running natively (spack-loaded g4beamline on AL9)") + print(f" events_per_job={events_per_job}, first_event={first_event}") + print(f" histo_file={histo_path}") + print(f" log_file={log_path}") + + # Stream g4bl stdout/stderr to BOTH the runner's stdout AND the SAM log + # file. Real-time visibility for the operator + persisted log for SAM push. + proc = subprocess.Popen(cmd_list, stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, text=True, bufsize=1) + with open(log_path, 'w') as log_f: + for line in proc.stdout: + log_f.write(line) + sys.stdout.write(line) + sys.stdout.flush() + proc.stdout.close() + rc = proc.wait() + + return jobdesc_entry['outputs'], histo_file, log_file, (rc == 0) + + def push_output(output_specs, output_file="output.txt", parents_file="parents_list.txt", simjob_setup=None): """ Generic function to push output files. @@ -700,31 +792,48 @@ def push_data(outputs, infiles, simjob_setup=None, track_parents=True): # Use generic push function return push_output(output_specs, "output.txt", parents_field, simjob_setup=simjob_setup) -def push_logs(fcl, simjob_setup=None): +def push_logs(fcl=None, simjob_setup=None, log_file=None): """Handle log file management and submission. - + + Either pass `fcl` (log filename derived via replace_file_extensions, the + art-side convention) or `log_file` directly (g4bl runner provides the SAM + name explicitly). At least one must be set. + Args: - fcl: FCL filename to derive log filename from - simjob_setup: Path to SimJob setup script for art environment + fcl: FCL filename to derive log filename from (art convention). + simjob_setup: Path to SimJob setup script for art environment. + log_file: Explicit log filename. Wins over `fcl` if both given. The + g4bl path uses this since there's no FCL. """ import shutil - - logfile = replace_file_extensions(fcl, "log", "log") - - # Copy jobsub log if available + + if log_file is not None: + logfile = log_file + elif fcl is not None: + logfile = replace_file_extensions(fcl, "log", "log") + else: + print("Warning: push_logs called with neither fcl nor log_file; nothing to push") + return 0 + + # Copy jobsub log if available (only meaningful when we derived from fcl + # and JOBSUB_LOG_FILE is the canonical source; for explicit log_file the + # runner has already streamed to it). jsb_tmp = os.getenv("JSB_TMP") - if jsb_tmp: + if jsb_tmp and log_file is None: src = os.path.join(jsb_tmp, "JOBSUB_LOG_FILE") print(f"Copying jobsub log from {src} to {logfile}") try: shutil.copy(src, logfile) except FileNotFoundError: print(f"Warning: Jobsub log not found at {src}") - + # Push log if it exists if Path(logfile).exists(): - output_specs = [("disk", logfile, "parents_list.txt")] - return push_output(output_specs, "log_output.txt", "parents_list.txt", simjob_setup=simjob_setup) + # G4bl jobs have no SAM-registered parents → parents_file="none". + # Art jobs use parents_list.txt (written by push_data earlier). + parents = "none" if log_file is not None else "parents_list.txt" + output_specs = [("disk", logfile, parents)] + return push_output(output_specs, "log_output.txt", parents, simjob_setup=simjob_setup) else: print(f"Warning: Log file {logfile} not found, skipping log push") return 0 \ No newline at end of file diff --git a/utils/runmu2e.py b/utils/runmu2e.py index 65e7093..7895d6c 100755 --- a/utils/runmu2e.py +++ b/utils/runmu2e.py @@ -16,33 +16,50 @@ process_template, process_direct_input, process_jobdef, + process_g4bl_jobdef, + build_mu2e_cmd, push_data, - push_logs + push_logs, ) -def main(): - parser = argparse.ArgumentParser(description="Execute production jobs from job definitions.") - parser.add_argument("--copy-input", action="store_true", help="Copy input files using mdh") - parser.add_argument('--dry-run', action='store_true', help='Print commands without actually running pushOutput') - parser.add_argument('--nevts', type=int, default=-1, help='Number of events to process (-1 for all events, default: -1)') - parser.add_argument('--mu2e-options', type=str, default='', help='Extra options to pass to mu2e command (e.g., "--no-timing --debug")') - parser.add_argument('--jobdesc', required=True, help='Path to the job descriptions JSON file (e.g., jobdefs_list.json)') - - args = parser.parse_args() - # Load and validate job descriptions from JSON file - with open(args.jobdesc, 'r') as f: - jobdesc = json.load(f) - - mode = validate_jobdesc(jobdesc) +def _dispatch_and_execute(mode, jobdesc, fname, args): + """Dispatch on runner mode, prep, execute, push. Returns True iff the + execute step failed (so main can exit nonzero). - # Get job definition by index from fname environment variable - fname = os.getenv("fname") - if not fname: - print("Error: fname environment variable is not set.") - sys.exit(1) + Encapsulates the per-runner asymmetry in one place: + - art runners (template / direct_input / normal) return an FCL which + this function then executes via `mu2e -c`. + - g4bl runner executes inside `process_g4bl_jobdef`; this function + only pushes its outputs. + """ + # G4Beamline: process_g4bl_jobdef both prepares and executes; it + # streams stdout to a SAM-named .log file. Push data only on success + # but always push the log if it exists, so failed grid jobs are + # debuggable in SAM. + if mode == 'g4bl': + try: + outputs, _histo_file, log_file, succeeded = process_g4bl_jobdef(jobdesc[0], fname, args) + except RuntimeError as e: + print(f"=== g4bl prep failed: {e} ===") + return True + + if not succeeded: + print("=== g4bl execution failed ===") + + if args.dry_run: + print("[DRY RUN] Would run: pushOutput output.txt") + else: + if succeeded: + push_data(outputs, infiles="", simjob_setup=None, track_parents=False) + else: + print("g4bl failed - skipping data push, attempting log push") + if log_file and Path(log_file).is_file(): + push_logs(log_file=log_file, simjob_setup=None) - # Process job based on mode + return not succeeded + + # Art runners: prep returns FCL; execute `mu2e -c` here. inloc = None # populated by process_jobdef; None for template/direct_input if mode == 'template': fcl, simjob_setup = process_template(jobdesc[0], fname) @@ -58,29 +75,21 @@ def main(): # on the push. All other cases (including None for template / direct # input) default to tracking parents. track_parents = not (isinstance(inloc, str) and inloc.startswith('dir:')) - - setup_cmd = f"source {simjob_setup}" - mu2e_cmd = f"mu2e -c {fcl}" - if args.nevts > 0: - mu2e_cmd += f" -n {args.nevts}" - if args.mu2e_options.strip(): - mu2e_cmd += f" {args.mu2e_options.strip()}" - - combined_cmd = f"{setup_cmd} && {mu2e_cmd}" - print(f"Executing: {combined_cmd}") + + cmd = build_mu2e_cmd(fcl, simjob_setup, args) + print(f"Executing: {cmd}") print(f"Working dir: {os.getcwd()}, FCL exists: {os.path.exists(fcl)}") - print("=== Starting Mu2e execution ===") + job_failed = False try: - run(combined_cmd, shell=True) + run(cmd, shell=False) print("=== Mu2e execution completed successfully ===") except subprocess.CalledProcessError as e: job_failed = True print(f"=== Mu2e execution failed with exit code {e.returncode} ===") - # Don't re-raise - we still want to upload logs and outputs - - # Handle output files and submission (even if job failed) + # Don't re-raise — we still want to upload logs and outputs + if not args.dry_run: if not job_failed: push_data(outputs, infiles, simjob_setup=simjob_setup, track_parents=track_parents) @@ -90,10 +99,32 @@ def main(): push_logs(fcl, simjob_setup=simjob_setup) else: print("[DRY RUN] Would run: pushOutput output.txt") - - # Exit with error code if job failed - if job_failed: + + return job_failed + + +def main(): + parser = argparse.ArgumentParser(description="Execute production jobs from job definitions.") + parser.add_argument("--copy-input", action="store_true", help="Copy input files using mdh") + parser.add_argument('--dry-run', action='store_true', help='Print commands without actually running pushOutput') + parser.add_argument('--nevts', type=int, default=-1, help='Number of events to process (-1 for all events, default: -1)') + parser.add_argument('--mu2e-options', type=str, default='', help='Extra options to pass to mu2e command (e.g., "--no-timing --debug")') + parser.add_argument('--jobdesc', required=True, help='Path to the job descriptions JSON file (e.g., jobdefs_list.json)') + + args = parser.parse_args() + + with open(args.jobdesc, 'r') as f: + jobdesc = json.load(f) + mode = validate_jobdesc(jobdesc) + + fname = os.getenv("fname") + if not fname: + print("Error: fname environment variable is not set.") sys.exit(1) + if _dispatch_and_execute(mode, jobdesc, fname, args): + sys.exit(1) + + if __name__ == "__main__": main() diff --git a/wiki/index.md b/wiki/index.md index 99f3894..af9d629 100644 --- a/wiki/index.md +++ b/wiki/index.md @@ -14,9 +14,11 @@ ### Sources -- [[pbi-sequence-workflow]] — how to use `json2jobdef` + `runmu2e` to produce `dts.mu2e.PBI*.art` outputs _(2026-04-21)_ +- [[pbi-sequence-workflow]] — full PBI chain (stage 1 dts → stage 2 mix dig → stage 3 reco mcs) via `json2jobdef` + `runmu2e` _(2026-04-25)_ - [[input-data-dir-shape]] — use `inloc: "dir:"` for cvmfs-resident inputs; basenames in input_data, runtime resolves via existing `dir:` prefix _(2026-04-21)_ - [[input-data-chunk-mode]] — `chunk_lines` input_data shape; on-the-fly chunking at grid time via `tbs.chunk_mode` + runmu2e sed slice. Best of split_lines and dir: without the trade-offs _(2026-04-22)_ +- [[metacat-reference]] — samweb→metacat CLI bridge, MQL patterns, Python API snippets, read-only MCP install (from `Mu2e/aitools`) _(2026-04-24)_ +- [[poms-reference]] — POMS data model (Campaign/Stage/Submission), dispatch lifecycle, Mu2e conventions (`i` naming via mkidxdef, dropbox path, decoupling possibility), `poms_client` library, common pitfalls _(2026-04-28)_ ### Analyses @@ -24,3 +26,4 @@ ### Maintenance - [[lint-2026-04-21]] — initial lint; wiki freshly initialized, 0 errors, 0 warnings, 1 info (coverage gap: no sources ingested yet) _(2026-04-21)_ - [[lint-2026-04-22]] — post-PBI-sequence lint; 0 errors, 2 warnings (raw-slug ambiguity, stale overview questions), 3 info _(2026-04-22)_ +- [[lint-2026-04-24]] — post-stage-2-mix lint; 0 errors, 4 warnings (raw-slug ambiguity recurring, 2 orphans, 1 stale overview claim), 4 info _(2026-04-24)_ diff --git a/wiki/log.md b/wiki/log.md index 2994235..cbf900d 100644 --- a/wiki/log.md +++ b/wiki/log.md @@ -82,3 +82,84 @@ Source: in-session revert 2026-04-21 in /tmp/pbi_dir.* ## [2026-04-22] lint | 0 errors, 2 warnings, 3 info Report: [[lint-2026-04-22]] Fixed: (A) dropped two stale Open Questions from overview.md + refreshed Current Understanding to reflect chunk_mode as canonical; (C) added [[input-data-chunk-mode]] cross-link in input-data-dir-shape.md Related section. Deferred: (B) raw-slug ambiguity — left as convention debate; (D) index annotation — ADR entry still accurate for the decision recorded. + +## [2026-04-24] ingest | Mu2e/aitools skills + MCP READMEs +Source: https://github.com/Mu2e/aitools (fetched 2026-04-24, 22 commits upstream). Pulled: skills/finding-data-metacat, skills/coding-with-metacat, mcp/metacat/README, mcp/sim-epochs/README. Synthesized as [[metacat-reference]] — prodtools-focused cheatsheet covering samweb→metacat CLI translation, MQL patterns, Python API snippets, read-only safety default, and install steps for metacat + sim-epochs MCP servers. Raw snapshot at [[2026-04-24-mu2e-aitools-skills]]. Skipped: SAM skill (internal knowledge), query-engine skill (not prodtools concern), building-* skills, dqm/code-index MCPs. + +## [2026-04-24] update | mix stage-2 push + POMS completion + event_id_per_index verified +Production push of `cnf.mu2e.PBI{Normal,Pathological}_33344Mix1BB.MDC2025ai_best_v1_3.0.tar` completed via `/mu2epro-run` with `--prod`; POMS map `MDC2025-025.json` extended in place (4 jobdef tarballs, 104 jobs total); SAM index `iMDC2025-025` recreated. Grid turnaround ~1 hour (cnf declared 05:00 UTC → dig datasets 06:00 UTC). All 52 mix jobs succeeded on first dispatch. Sample file metadata confirms `event_id_per_index` produced globally unique `(run, subrun, event)` tuples as designed — index 21 → (1430, 21, 21001..22000), matching the offset+step*index formula. Updated [[pbi-sequence-workflow]] Stage 2 → "Production push / POMS run" subsection. + +## [2026-04-24] commission | metacat-readonly MCP server +Installed `Mu2e/aitools/mcp/metacat` under `muse_050125/aitools/`. Venv requires Python 3.10+ (Mu2e ops env; system py3.9 fails `mcp>=1.2.0`). Registered project-level via new `.mcp.json` + `.claude/settings.json enabledMcpjsonServers: ["metacat-readonly"]`. All 4 tools (`discover_datasets`, `get_dataset_details`, `query_dataset_files`, `get_server_info`) verified against live metacat on 2026-04-24. Schema quirk noted: `sort_by` limited to fixed short-name set (not arbitrary metadata keys). Updated [[metacat-reference]] with commissioned status + install recipe + schema quirks. + +## [2026-04-24] lint | 0 errors, 4 warnings, 4 info +Report: [[lint-2026-04-24]] +Inventory: 8 pages, 2 raw sources, 11 distinct wikilinks. Warnings: (1) raw-slug ambiguity recurring — `[[2026-04-21-pbi-sequence-implementation]]` + `[[2026-04-24-mu2e-aitools-skills]]` resolve only in `raw/`, deferred per [[lint-2026-04-22]]; (2) [[lint-2026-04-22]] orphan — closed by today's lint referencing it; (3) [[metacat-reference]] orphan — only inbound from index.md and raw, no cross-link from [[pbi-sequence-workflow]] Stage 2 despite shared subject; (4) stale claim in `overview.md` Open Questions about chunk_mode scale — N=52 successful run on 2026-04-22 contradicts the framing. Info: raw-frontmatter convention not formalized; `Campaigns`/`Incidents` index sections still empty; `pbi-sequence-workflow ↔ metacat-reference` cross-link gap; lint-chain hygiene noted. Fixed: none yet — awaiting user confirmation on which warnings to apply. + +## [2026-04-25] update | pbi-sequence-workflow Stage 3 reco verified end-to-end +Reason: added Stage 3 (`dig → mcs`) reco entry to `data/mdc2025/reco.json` matching the ag pattern (`tarball_append: "-reco"`, array cross-product). 1-event smoke test on PBINormal index 0 passed (exit 0, all reco modules KKDe/Dmu/Ue/Umu + helix + calo + crv ran with Visited=1 Passed=1; 568 KB mcs output preserving the per-index sequencer `001430_00000021`). Per-index event chain `dig→mcs` confirmed: index 21 → events 21001..22000 carry through. Non-obvious gotcha surfaced: reco.json entries need explicit `services.DbService.{purpose,version}` overrides or jobs fail at `ProtonBunchTimeFromStrawDigis` with `EMPTY -1/-1/-1` calibration set; existing af/ag entries lack these, possibly never smoke-tested locally — flagged in workflow page as open question. PBIPathological smoke + production push pending. +Source: in-session test 2026-04-25 under `oksuzian` user; tarballs in repo root. + +## [2026-04-25] update | pbi-sequence-workflow Stage 3 PBIPathological smoke verified +Reason: second of two PBI flavors confirmed working — `dig.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.art` index 0 reco passes (exit 0, all modules Visited=1 Passed=1, CPU 1.55s, VmPeak 1.98 GB). Both PBI flavors validated locally under MDC2025ai env. Stage 3 section in [[pbi-sequence-workflow]] updated with Pathological smoke result. Production push (`/mu2epro-run --prod`) is the only remaining step. +Source: in-session test 2026-04-25. + +## [2026-04-25] update | pbi-sequence-workflow Stage 3 reco production push complete +Reason: `/mu2epro-run MDC2025ai json2jobdef --json data/mdc2025/reco.json --dsconf MDC2025ai_best_v1_3 --prod --jobdefs MDC2025-026.json` completed at 11:24 UTC. Both reco tarballs (`cnf.mu2e.PBI{Normal,Pathological}_33344Mix1BB-reco.MDC2025ai_best_v1_3.0.tar`) SAM-declared and copied to `/pnfs/.../phy-etc/cnf/mu2e/PBI*Mix1BB-reco/MDC2025ai_best_v1_3/tar/...`. New POMS map `MDC2025-026.json` (52 jobs total, 26 per flavor); SAM index `iMDC2025-026` def_id 218067 declared. POMS scan will pick up the 52 reco jobs next pass. Expected mcs outputs: `mcs.mu2e.PBI{Normal,Pathological}_33344Mix1BB.MDC2025ai_best_v1_3.art` (26 files each) + sibling logs. Full PBI chain (dts → dig → mcs) now in production. +Source: /mu2epro-run skill output 2026-04-25 06:24 CDT; verified via `samweb list-files` + `samweb describe-definition iMDC2025-026`. + +## [2026-04-25] update | pbi-sequence-workflow Stage 3 push remediation (POMS map convention) +Reason: initial Stage 3 push targeted a fresh `MDC2025-026.json` — wrong; PBI chain stages 1+2 already in `MDC2025-025`, the convention is extend-in-place. Remediated by re-running `json2jobdef --prod --jobdefs MDC2025-025.json` (tarballs already in SAM, pushOutput no-op; entries appended). `MDC2025-025.json` now 6 entries / 156 jobs total (Stage 1 × 2 + Stage 2 × 2 + Stage 3 × 2). `iMDC2025-025` regenerated by `mkidxdef --prod`, def_id 218087, dimension `dh.sequencer < 0000156`. Orphan `MDC2025-026.json` map file and `iMDC2025-026` SAM index deleted. Saved feedback memory `feedback_extend_existing_poms_map.md` so the convention is auto-loaded next session. samweb-CLI quirk noted: `delete-definition iMDC2025-026` hit `RecursionError` under ksu mu2epro from one shell, succeeded from another — root cause not investigated. +Source: in-session 2026-04-25 ~11:53 UTC. + +## [2026-04-25] commission | poms-push skill +Drafted `.claude/commands/poms-push.md` to codify the extend-vs-allocate POMS map decision (the convention I broke earlier today by allocating MDC2025-026 instead of extending 025). Behavior: read JSON config + dsconf, derive workflow pattern by stripping known stage suffixes (Mix1BB, -reco, Triggered, etc.) and taking longest common prefix across entries; scan `^MDC2025-\d{3}\.json$` (filter excludes -test/-tes/-MDS3c variants in one regex), count matching tarballs in each; print recommended `/mu2epro-run` invocation and stop (does NOT push — production gate stays in /mu2epro-run). Validated on canonical case `data/mdc2025/reco.json --dsconf MDC2025ai_best_v1_3 MDC2025ai`: derived pattern PBI, found 6 matching tarballs in MDC2025-025, decided "extend in place" — exactly the post-remediation correct answer. Skill is registered and visible in the skill list. Convention is now triple-anchored: memory `feedback_extend_existing_poms_map.md` (auto-loaded), wiki Stage 3 "Process note" (queryable rationale), `/poms-push` skill (executable enforcement at decision time). +Source: in-session 2026-04-25 ~12:10 UTC. + +## [2026-04-25] update | listNewDatasets gains --completeness flag with auto-rebuild +Reason: completeness questions previously required pomsMonitor (campaign-scoped) or manual `jobquery --njobs`/`samweb count-files` per dataset. Added `--completeness` to `listNewDatasets` that joins against the existing pomsMonitor SQLite DB (`/poms_data.db`) and prints `/` per row. Includes auto-rebuild: cheap mtime check against POMS map files in lookback window; if any map newer than DB, run `build_db(since=now-days)` to refresh only changed entries. `--no-rebuild` opts out. Verified end-to-end on the freshly-completed PBI Stage 3 mcs datasets — both 26/26 complete; fast path (DB fresh) sub-second; rebuild path ~190s for the one stale map. Quirk: requires `pyenv ana` post `muse setup ops` for SQLAlchemy import — saved as `reference_pyenv_ana_for_db.md` memory. +Source: in-session 2026-04-25 ~12:30 UTC. + +## [2026-04-25] update | pbi-sequence-workflow Stage 3 reco completed in production +Reason: POMS dispatched and completed all 52 PBI reco jobs (Normal + Pathological, 26 each) within ~30 minutes of the SAM index recreation. Verified via `listNewDatasets --completeness`: both mcs datasets show 26/26. Sibling log datasets also landed. Full PBI chain (dts → dig → mcs) end-to-end in production with globally-unique (run, subrun, event) tuples preserved. +Source: listNewDatasets query 2026-04-25 ~12:30 UTC. + +## [2026-04-25] update | bin wrappers guard SQLAlchemy import +Reason: cryptic ModuleNotFoundError tracebacks turned into clear "Run pyenv ana" message at startup. Added to `bin/pomsMonitor`, `bin/list_no_child_datasets`, `bin/pomsMonitorWeb` (also guards Flask) — exit 2 on missing module. `bin/listNewDatasets` (via `utils/listNewDatasets.py`) checks only when `--completeness` is requested and degrades softly: prints warning, disables completeness column, runs the rest. Verified both modes by running each wrapper in `env -i` shell with only `muse setup ops` (no `pyenv ana`). Memory `reference_pyenv_ana_for_db.md` updated to describe the new clear symptom. +Source: in-session 2026-04-25 ~12:50 UTC. + +## [2026-04-25] commission | recent-datasets skill +Drafted `.claude/commands/recent-datasets.md` to wrap `bin/listNewDatasets --completeness` with the right env (Mu2e setup + `pyenv ana` for SQLAlchemy + `python3` not `bash`) and sensible defaults (`--days 1`, completeness on). Also filters the noisy db_builder rebuild trace lines (`Skipping logparser ...`, `Loaded N job definitions`, `Discovered and cached ...`, etc.) so output is just the dataset table — keeps real signals (DB stale messages, warnings, custom-query echo) intact. Encodes three frictions hit during this session: wrong invocation (bash vs python3), forgot pyenv ana, forgot --completeness. Verified pipeline produces clean table on PBI mcs query (both 26/26). +Source: in-session 2026-04-25 ~13:00 UTC. + +## [2026-04-27] commission | parallel-audit skill +Drafted `.claude/commands/parallel-audit.md` to encode the "fan out N Explore agents on non-overlapping slices, then synthesize" pattern. Inspired by Hermes Agent's `software-development/subagent-driven-development` skill (NousResearch/hermes-agent). Behavior: parse ` [--agents N]` (default 4, clamp [2,6]); pick non-overlapping slicing dimension (by directory / concern / layer / risk); spawn all agents in a single tool-call message; synthesize returns into prioritized punch list with [critical|high|medium|low] tags and file:line citations. Dedupes findings (multi-agent mentions = higher confidence). Default Mu2e 4-cut documented in skill: `utils+bin code quality / CLI ergonomics+EXAMPLES drift / DB+JSON / tests+repo hygiene` — the cut already validated by today's deep-review run. Read-only by design; agents do not edit. Skill registered and visible in skill list. +Source: in-session 2026-04-27 ~14:00 UTC, after Hermes Agent comparison + earlier 4-agent prodtools audit. + +## [2026-04-27] update | repo hygiene + Mu2eFilename consolidation +Reason: multiple noise files at repo root (`os`, `sys` PostScript blobs ~8MB each, `test2/`, `test_runmu2e/`, `test_reco/`, `prompts*.txt`, `momentum_resolution_*.png`, `MDC*-test.json`, `mu2e_common.gdml`) cluttering `git status`; duplicate `Mu2eFilename` class in `utils/job_common.py:15` and `utils/datasetFileList.py:21`. Added 11 root-anchored entries to `.gitignore` (untracked count dropped ~50→~30). Removed local `Mu2eFilename` from `datasetFileList.py`, added `relpathname()` (SHA256 hash subdir, byte-identical to old) to canonical class in `job_common.py`. Existing 13 unit tests across `TestMu2eFilename` (8) and `TestDatasetFileListFilename` (5) became regression tests for the merge — full suite 181/181 passing. Caveat: canonical class enforces 6+ dot-separated fields and raises `ValueError`; old local class was lenient. Safe in current call site (`f` from samweb is always well-formed) but flagged for downstream callers. +Source: in-session 2026-04-27 ~13:30 UTC. + +## [2026-04-27] update | ~/.claude moved to /exp/mu2e/app +Reason: `/nashome` is at 93% capacity; `~/.claude` was 16M and growing. `~/.claude` is now a symlink to `/exp/mu2e/app/users/oksuzian/.claude` (cephfs, 334G free). Live session writes to `history.jsonl` continued through the symlink without disruption. Memory paths in MEMORY.md still resolve (the project memory key `-exp-mu2e-app-users-oksuzian-muse-050125-prodtools` is unchanged). Watch for any I/O lag on cephfs — small JSONL appends are the worst case but no symptoms so far. +Source: in-session 2026-04-27 ~13:45 UTC. + +## [2026-04-28] update | g4bl tarball pushed to production SAM +First g4bl tarball declared in production SAM via `pushOutput`. Hand-built `cnf.mu2e.G4blPOT.TESTaa.0.tar` (625KB, contains `jobpars.json` + `work/Mu2E.in` + `work/Geometry/*.txt`). Resolves Unknown #1 from the demonstrator plan: `pushOutput` accepts our minimal `jobpars.json` (runner/main_input/events_per_job/desc/dsconf — no FCL-derived metadata) and produces valid SAM declaration. Tarball lives at `/pnfs/mu2e/tape/phy-etc/cnf/mu2e/G4blPOT/TESTaa/tar/c7/74/cnf.mu2e.G4blPOT.TESTaa.0.tar` (hash subdirs from `Mu2eFilename.relpathname()`). Pushed via `ksu mu2epro` direct (not `/mu2epro-run` because that skill expects prodtools `bin/` scripts; `pushOutput` is a UPS binary on PATH after `setup OfflineOps`). Next: Step 2 (`mkidxdef` for the dummy index dataset), Step 3 (POMS map JSON to dropbox). +Source: in-session 2026-04-28 ~14:30 UTC. + +## [2026-04-28] ingest | poms-reference (FNAL POMS architecture + Mu2e conventions) +Spawned 3 parallel Explore agents to research github.com/fermitools/poms on architecture/data model, SAM-dataset/map wiring, and user operations. Synthesized into wiki/pages/poms-reference.md with raw sources at wiki/raw/2026-04-28-poms-{architecture,sam-wiring,user-ops}.md. Key findings: (1) POMS data model is Campaign → CampaignStage → Submission → Jobs, with the SAM-dataset name configured per-stage in POMS DB. (2) `i` is a Mu2e mkidxdef convention (in our `prod_utils.py:create_index_definition`), NOT a POMS hardcode — Agent #2 initially confused this; corrected in synthesis. (3) `poms_client` Python library exists with `update_campaign_stage()` and `launch_campaign_stage_jobs()` — stage config IS scriptable, not web-UI-only. (4) `iMDC2025-NNN` SAM defs are content-agnostic placeholders (just `etc.mu2e.index.000..txt` files for job-count iteration); reusable across stages. Confirmed by user 2026-04-28 in the context of testing g4bl runner. Open questions: exact column name for stage's SAM dataset in `campaign_stages`, exact `update_campaign_stage()` payload, whether running Mu2e POMS allows admin-free SAM-def reuse — flagged in page for verification on the running instance. Pages written: poms-reference. Pages updated: index.md. +Source: in-session 2026-04-28, three Explore agents on github.com/fermitools/poms. + +## [2026-04-28] update | iG4BL-000 SAM definition created +After samweb-write auth resolved upstream, retried `mkidxdef --prod` against the dropbox map `G4BL-000.json`. SAM def `iG4BL-000` (id 218203) created with one file `etc.mu2e.index.000.0000000.txt` (query `dh.dataset etc.mu2e.index.000.txt and dh.sequencer < 0000001`). Quirk: despite "Exceeded 30 redirects" error message during the create call, the operation actually succeeded — the def is registered under Username=`oksuzian` (not mu2epro, despite running through `ksu mu2epro`). Demonstrator state now: tarball at /pnfs/mu2e/tape/phy-etc/cnf/mu2e/G4blPOT/TESTaa/tar/c7/74/, map at /exp/mu2e/app/users/mu2epro/production_manager/poms_map/G4BL-000.json (5-field minimal shape), SAM def iG4BL-000 with 1 file. Ready for POMS-side stage configuration. +Source: in-session 2026-04-28 ~22:27 UTC. + +## [2026-04-28] update | g4bl runner switched from SL7 container to native AL9 spack +User noted that `source mu2e-art.sh && spack load g4beamline` works natively on AL9 — no SL7 container needed. Refactored `process_g4bl_jobdef`: removed apptainer wrap path entirely (-50 LOC), removed `_is_inside_sl7()` helper, removed `DEFAULT_G4BL_CONTAINER` constant. New runner just does `unset SPACK_ENV PYTHONHOME PYTHONPATH PYTHONNOUSERSITE; source mu2e-art.sh; eval "$(spack load --sh g4beamline)"; cd embed_dir; g4bl viewer=none First_Event=N Num_Events=M epsMax=0.01 histoFile=...`. Three discoveries: (1) g4bl 3.08b on AL9 (built against gcc-13.3.0 + Geant4 11.3.2) requires plain `key=value` CLI syntax, NOT `param key=value` — older 3.08 SL7 build was lenient. (2) `unset SPACK_ENV` is critical: `bin/runmu2e` does `muse setup ops` before invoking Python, which activates ops-019; subprocess inherits SPACK_ENV; `spack load g4beamline` then fails because g4beamline isn't in ops-019. Documented in reference_spack_env_after_muse_setup memory + Mu2e wiki. (3) Native AL9 path eliminates the entire SL7 nesting consideration; workers run in fnal-wn-el9 (standard fermigrid.cfg outer container) with no inner wrap. `poms/g4bl.cfg` could now be retired in favor of fermigrid.cfg. Both smoke modes pass: embed_dir produces 82KB ROOT + 570KB log; tarball mode produces 74KB ROOT + 570KB log; 0 fatal exceptions, 51 geometry warnings (normal for Mu2e Mau9 geometry). 181/181 unit tests still pass. +Source: in-session 2026-04-28 ~21:12 UTC, after user revealed `spack load g4beamline` works on AL9. + +## [2026-04-28] update | retired poms/g4bl.cfg +Deleted `poms/g4bl.cfg` (SL7 outer-container submit cfg). No longer needed after the runner switched to native AL9 spack g4bl — workers can now run under the standard `poms/fermigrid.cfg` (fnal-wn-el9). Updated `wiki/pages/poms-reference.md` to note the retirement. +Source: in-session 2026-04-28. diff --git a/wiki/pages/lint-2026-04-24.md b/wiki/pages/lint-2026-04-24.md new file mode 100644 index 0000000..9bcec4c --- /dev/null +++ b/wiki/pages/lint-2026-04-24.md @@ -0,0 +1,135 @@ +--- +title: Lint Report 2026-04-24 +tags: [lint, maintenance] +sources: [] +updated: 2026-04-24 +--- + +# Lint Report — 2026-04-24 + +Third lint pass; second since the wiki gained the metacat reference +and stage-2 PBI documentation. Predecessor: [[lint-2026-04-22]] (also +[[lint-2026-04-21]]). + +## Summary + +- 🔴 Errors: 0 +- 🟡 Warnings: 4 +- 🔵 Info: 4 + +Inventory: 8 pages, 2 raw sources, 11 distinct `[[slug]]` references +(after stripping illustrative `[[slug]]` / `[[...]]` markers in code +spans per [[lint-2026-04-21]] §Notes). + +## 🔴 Broken Links + +None at the strict pages/-only level. +(See 🟡 Raw-slug ambiguity below for the deferred convention question.) + +## 🔴 Missing Frontmatter + +None among `pages/`. All eight pages have the four required fields: +`title`, `tags`, `sources`, `updated`. + +## 🟡 Warnings + +### Raw-slug ambiguity (recurring) + +Two `[[slug]]` references resolve only to `raw/`, not `pages/`, which +SCHEMA.md §Cross-References defines as the only valid target: + +- [[pbi-sequence-workflow]] → `[[2026-04-21-pbi-sequence-implementation]]` +- [[pbi-sequence-workflow]] → (extends) raw source page; same shape +- [[2026-04-21-extend-jobdef-per-index-overrides]] → `[[2026-04-21-pbi-sequence-implementation]]` +- [[2026-04-21-fold-pbi-into-json2jobdef]] → (transitively via raw) +- [[metacat-reference]] → `[[2026-04-24-mu2e-aitools-skills]]` + +Strict reading: broken. Pragmatic reading: `[[slug]]` lookups should +fall back to `raw/.md` when `pages/.md` is absent. +Already deferred as a convention debate in [[lint-2026-04-22]]; +unchanged today. Recommendation: formalize the fallback in `SCHEMA.md` +(one-sentence amendment) so future lints stop flagging it. + +### Orphan: [[lint-2026-04-22]] + +No inbound `[[slug]]` references from any non-index page. Normal for +lint reports until the next lint references them. Fix: today's lint +references it (above) — closes the orphan. + +### Orphan: [[metacat-reference]] + +Inbound only from `index.md` and `raw/2026-04-24-mu2e-aitools-skills.md`. +[[pbi-sequence-workflow]] Stage 2 mentions the metacat MCP and the +sample-file metadata it returned but never wikilinks to +[[metacat-reference]]. Easy fix: add a cross-reference in the Stage 2 +"Validation status" or "Production push completed" subsections. + +### Stale claim in overview.md + +`wiki/overview.md` "Open Questions" section still asks: + +> Scale test of `chunk_mode` at N ≫ 52 jobs (first production run +> used chunk_lines=1000 on a ~52,000-line source). Does sed on the +> cvmfs source behave well when many workers hit it concurrently? + +The 2026-04-22 stage-1 aj production run executed 52 PBI sequence jobs +in `chunk_mode` against the same cvmfs source — successful on the +first POMS dispatch (verified via metacat query 2026-04-24, see +[[pbi-sequence-workflow]] → "POMS grid run completed"). N ≫ 52 is +still untested, but the N=52 data point is now positive. Suggest +refreshing the Open Question to: + +> Scale test of `chunk_mode` at N > 100 jobs. (N=52 verified on +> 2026-04-22; concurrent sed on cvmfs source produced no observable +> issues. Need a larger fan-out before declaring scaling resolved.) + +## 🔵 Info + +### Raw frontmatter convention not formalized + +`raw/2026-04-21-pbi-sequence-implementation.md` uses prose metadata +(`**Date:**`, `**Type:**`, `**Context:**` headers); `raw/2026-04-24-mu2e-aitools-skills.md` +uses YAML frontmatter (`title`, `source_type`, `source_url`, +`fetched`). Both are reasonable for "immutable source document"; SCHEMA +doesn't specify a shape. Worth picking one and noting it in SCHEMA.md +to keep future ingests consistent. No action required this lint. + +### No `Campaigns` / `Incidents` entries in `index.md` + +Both index sections still hold only the placeholder +``. Today's PBI mix push (which +verified the full MDC2025aj→ai chain end-to-end and uncovered the +`mu2e-trig-config` schema drift in aj) is candidate material for both: + +- **Campaign:** `MDC2025-025-pbi-mix-2026-04-24` — first prodtools-driven + PBI sequence + mix push, end-to-end verified. +- **Incident:** `mdc2025aj-trig-config-schema-drift` — `FilterEcalNNTrigger` + pset mismatch in shipped Musings, blocks aj mix until upstream + refresh. (Already captured in [[pbi-sequence-workflow]] Gotchas; an + Incident page would surface it independently of the workflow page.) + +Not blocking. + +### Cross-reference gap: [[pbi-sequence-workflow]] ↔ [[metacat-reference]] + +Both pages discuss the metacat MCP server and how it surfaces stage-2 +mix outputs. Each one stands on its own; adding a `[[metacat-reference]]` +link from Stage 2 (and back-link from `metacat-reference` to +[[pbi-sequence-workflow]] — already present) closes the cycle. + +### Lint-chain hygiene + +Today's lint references both predecessors ([[lint-2026-04-21]] and +[[lint-2026-04-22]]); previous lints had only one back-link each. +Continue this convention: every lint should reference at least its +direct predecessor, ideally the full chain. + +## Notes + +- `[[slug]]` and `[[...]]` inside inline code or fenced blocks are + illustrative examples (cross-reference syntax in SCHEMA.md, prior + lint reports). Lint ignores them. +- Pages in this lint inventory: 8 (`pages/*.md`) + 2 (`raw/*.md`). + Total wiki size: ~1718 lines. +- Latest activity: heavy edits 2026-04-23/24 (stage-2 PBI mix push, + metacat MCP commissioning, this lint). diff --git a/wiki/pages/metacat-reference.md b/wiki/pages/metacat-reference.md new file mode 100644 index 0000000..9ac944b --- /dev/null +++ b/wiki/pages/metacat-reference.md @@ -0,0 +1,239 @@ +--- +title: Metacat reference for prodtools +tags: [reference, metacat, data-handling, mcp, commissioned] +sources: [2026-04-24-mu2e-aitools-skills] +updated: 2026-04-24 +--- + +# Metacat reference for prodtools + +Pulled from [`Mu2e/aitools`](https://github.com/Mu2e/aitools) skills +`finding-data-metacat` + `coding-with-metacat` and the MCP-server +READMEs. Kept tight to prodtools use cases — catalog lookups we today +do via `samweb`, plus programmatic access for `bin/`-side scripts. + +## When to use metacat + +- SAM is winding down; metacat is the successor catalog. New workflows + should prefer metacat. +- For existing prodtools paths that already call `samweb` (pushout, + runmu2e, listNewDatasets, etc.), don't force-switch until there's a + concrete benefit. +- `/mu2epro-run` + `pushOutput` still declares to SAM today. The + transition plan isn't something prodtools controls. + +## samweb → metacat CLI translation + +Quick bridge for the commands we use most. + +| samweb | metacat | +|---|---| +| `samweb list-definitions --defname ''` | `metacat dataset list ''` | +| `samweb count-files 'dh.dataset='` | `metacat query "files from :" \| wc -l` | +| `samweb describe-definition ` | `metacat dataset show :` | +| `samweb list-files 'file_name in (...)'` | `metacat query "files from : where name in (...)"` | +| `samweb locate-file ` | `mdh print-url -l tape -s xrootd ` | + +Datasets/files use **DIDs**: `:` (e.g. +`mu2e:dts.mu2e.PBINormal_33344.MDC2025aj.art`). + +## Environment setup + +```bash +mu2einit # or source setupmu2e-art.sh +muse setup ops +getToken # OAuth token (~2h), if needed +metacat auth login -m token $USER # session auth +metacat auth whoami # verify +``` + +Most prodtools shells already have `metacat` on PATH after +`muse setup ops`. + +## MQL query patterns + +Metacat Query Language. **Not SQL** — the keyword is `files from`, not +`select`. `query()` in Python returns a lazy generator; wrap with +`list(...)` to force evaluation. + +```bash +# Basic: all files in a dataset +metacat query "files from mu2e:dts.mu2e.PBINormal_33344.MDC2025aj.art" + +# Subset: by subrun/run +metacat query "files from $DS where rs.first_subrun > 100" +metacat query "files from $DS where run = 1430" + +# Filter by tier + size +metacat query "files where dh.tier=sim and file_size > 1000000000" + +# Multi-dataset union +metacat query "files from mu2e:dataset1.art, mu2e:dataset2.art" + +# From a file +metacat query -q my_query.mql +``` + +Clauses supported: `files from `, `where `, `order by +`, `limit `, `offset `. + +## Python API (for prodtools bin scripts) + +```python +from metacat.webapi import MetaCatClient +client = MetaCatClient() # env-driven server/token +``` + +**Core read-only methods** (no auth required for reads): + +| Method | Use | +|---|---| +| `list_datasets(namespace_pattern=, with_counts=False)` | enumerate datasets | +| `get_dataset(did=, exact_file_count=False)` | dataset metadata + file_count | +| `get_dataset_files(did=, with_metadata=False)` | files in dataset | +| `get_file(did=, with_metadata=True)` | single file; returns `None` if missing (does not always raise) | +| `query(mql_string)` | MQL query; **lazy generator** — wrap with `list(...)` | + +**Scalable pattern — filter first, then fetch counts:** + +```python +# DON'T: with_counts=True across all datasets (slow) +# DO: +datasets = client.list_datasets(namespace_pattern="mu2e", with_counts=False) +selected = [d for d in datasets if d["name"].startswith("dts.mu2e.PBI")] +for d in selected: + did = f"{d['namespace']}:{d['name']}" + info = client.get_dataset(did=did, exact_file_count=False) + print(did, info.get("file_count", 0)) +``` + +**Pagination for large result sets:** + +```python +limit, offset, out = 100, 0, [] +while True: + batch = list(client.query(f"files from {did} limit {limit} offset {offset}")) + if not batch: break + out.extend(batch); offset += limit +``` + +**Generate xroot URLs from a metacat query** (replaces samweb+locate +in `bin/` scripts): + +```python +import subprocess +for f in client.query(f"files from {did} where run = {run}"): + fdid = f"{f['namespace']}:{f['name']}" + url = subprocess.run(["mdh", "print-url", "-l", "tape", "-s", "root", fdid], + capture_output=True, text=True).stdout.strip() + print(url) +``` + +## Safety: read-only default + +Upstream skill convention for AI-generated code: **default to read-only +methods**; require explicit user approval before calling any of +`declare_*`, `create_*`, `update_*`, `add_*`, `remove_*`, `delete_*`, +`retire_*`. prodtools inherits this — we don't write to the catalog +from bin/ today. The `pushOutput` path that registers outputs is +handled by `OfflineOps`, not by metacat Python API. + +Upstream ships a `SafeMetaCat` wrapper class (in +`coding-with-metacat/SKILL.md`) that intercepts write methods and +raises unless `ALLOW_WRITES=True`. Worth adopting if we start +generating any declare/update code. + +## Namespace & file naming (recap) + +``` +:..... # file DID +:.... # dataset DID (no sequencer) +``` + +Matches the Mu2e 6-field convention we already use: `dts.mu2e....art`. + +## MCP servers available from Mu2e/aitools + +`Mu2e/aitools/mcp/` ships three MCP servers. Relevant to prodtools: + +### `metacat` (read-only) — commissioned 2026-04-24 + +Four tools: `discover_datasets`, `get_dataset_details`, +`query_dataset_files`, `get_server_info`. + +**Status:** installed and running in the prodtools project. Venv at +`/exp/mu2e/app/users/oksuzian/muse_050125/aitools/mcp/metacat/.venv` +(Python 3.10.14 from Mu2e ops env — system Python 3.9 is too old for +the `mcp>=1.2.0` package, so build the venv after +`source setupmu2e-art.sh && muse setup ops`, not from bare system +python3). `scripts/start_mcp.sh` sources the Mu2e env internally, so +the MCP launcher needs no shell setup at call time. + +Project-level registration (already in place): + +- `.mcp.json` at repo root — defines `mcpServers.metacat-readonly` with + absolute command path to `scripts/start_mcp.sh`. +- `.claude/settings.json` — adds `enabledMcpjsonServers: + ["metacat-readonly"]` so the server auto-approves on session start. + +**Install recipe** (for a fresh machine): + +```bash +cd +git clone https://github.com/Mu2e/aitools.git +source /cvmfs/mu2e.opensciencegrid.org/setupmu2e-art.sh +muse setup ops +cd aitools/mcp/metacat +python3 -m venv --system-site-packages .venv +source .venv/bin/activate +pip install -U pip -r requirements.txt -e . +bash scripts/start_mcp.sh --check # smoke test +``` + +Auth: uses env-provided metacat config (`MetaCatClient()`), no +MCP-layer auth handling. Needs `getToken` / `metacat auth login` to +have been run. + +**Verified in production (2026-04-24):** all 4 tools respond correctly +against live SAM/metacat data. Used to confirm the PBI mix pipeline +(see [[pbi-sequence-workflow]] → "POMS grid run completed"). + +### Schema quirks for the MCP tools + +- `query_dataset_files` `sort_by` is limited to a fixed field set: + `['created_timestamp', 'n_events', 'name', 'run', 'size', + 'subrun']`. Passing full metadata keys like `rs.first_subrun` is + rejected with `sort_by must be one of: [...]`. The short names are + conveniently mapped to the underlying metadata paths — no way to + sort by arbitrary metadata keys in the current MCP API. +- `discover_datasets` `name_pattern` uses shell-style globs (`*`, `?`) + — same as the samweb `--defname` flag, not MQL `like` syntax. +- `include_metadata=true` on `discover_datasets` returns empty + `metadata: {}` for datasets; per-file metadata is accessed via + `get_dataset_details` (sample file) or + `query_dataset_files(include_metadata=true)`. + +### `sim-epochs` (JSON catalog over MDC2025xx epochs) + +Two tools: `get_simulation_epochs()`, `get_datasets_for_epoch(epoch)`. +File-backed (`data/sim_catalog.json`, overridable via +`SIM_EPOCHS_FILE`). Potentially useful for MDC2025aj/ai/ag epoch +queries without a round-trip to SAM/metacat. Install recipe: same +pattern as metacat, under `aitools/mcp/sim-epochs/`. + +### `dqm` (not currently used by prodtools) + +DQM Query Engine read-only MCP. Skipped. + +## Gaps worth filling upstream + +`Mu2e/aitools` currently has no prodtools/POMS/jobdef skills. The +content in `wiki/pages/pbi-sequence-workflow.md` (Stage 1+2), +`input-data-*` shapes, and our POMS push flow would fit naturally as +upstream skills. Low-effort PR; they have 4 contributors. + +## Related + +- [[pbi-sequence-workflow]] — production chain this reference supports +- [[input-data-chunk-mode]], [[input-data-dir-shape]] — input_data shapes +- Source: [[2026-04-24-mu2e-aitools-skills]] diff --git a/wiki/pages/pbi-sequence-workflow.md b/wiki/pages/pbi-sequence-workflow.md index 5ca4212..2e8e226 100644 --- a/wiki/pages/pbi-sequence-workflow.md +++ b/wiki/pages/pbi-sequence-workflow.md @@ -1,8 +1,8 @@ --- title: PBI sequence generation workflow -tags: [reference, workflow, primary-generation] +tags: [reference, workflow, primary-generation, mixing, reco] sources: [2026-04-21-pbi-sequence-implementation] -updated: 2026-04-21 +updated: 2026-04-25 --- # PBI sequence generation workflow @@ -160,8 +160,9 @@ PBI objects into art). 26 jobs runs in minutes locally. 4. `create_jobdef` detects `source.module_type: PBISequence` in the FCL, applies the `PBISequence` validation + tbs construction branch: sets `tbs.inputs` (fileNames list), `tbs.event_id` - (runNumber only), `tbs.subrunkey = ""` (no per-job subrun — rejected - by PBISequence's pset validator). + (runNumber only), `tbs.subrunkey = ""` (no per-job subrun by default; + per-index overrides via `event_id_per_index` are supported on + MDC2025aj+, see Gotchas → "Update 2026-04-22"). 5. At job time, `jobfcl --index N` picks `fileNames[N]` from the list and emits FCL with the chunk's basename. Runtime resolves the basename via `--default-location dir:/`. @@ -171,10 +172,11 @@ PBI objects into art). 26 jobs runs in minutes locally. - **Chunk files are written locally.** If you run the jobs on the grid, the chunk text files need to be accessible from the grid node (stash or resilient dCache). For local-only runs this is fine. -- **Subrun is the same across chunks.** Output uniqueness comes from - the input chunk basename's sequencer slot (`.00`, `.01`, ...), not - from per-job subruns. PBISequence's pset validator rejects - `source.firstSubRunNumber` (see Gotchas). +- **Subrun is the same across chunks** (pre-MDC2025aj). Output uniqueness + comes from the input chunk basename's sequencer slot (`.00`, `.01`, ...), + not from per-job subruns. PBISequence's pset validator rejected + `source.firstSubRunNumber` up to MDC2025ai; MDC2025aj accepts it (see + Gotchas → "Update 2026-04-22"). ## Gotchas discovered 2026-04-21 @@ -195,7 +197,10 @@ Default jobfcl behavior treats `source.fileNames` entries as SAM-known dataset files, which 404s for our chunks (they're not in SAM). Use `dir:` to route through the local filesystem. -### PBISequence pset validator rejects common source parameters +### PBISequence pset validator rejects common source parameters (pre-MDC2025aj) + +**Applies to MDC2025ai and earlier.** Superseded in part by MDC2025aj; +see "Update 2026-04-22" below. The PBISequence C++ module accepts only: `fileNames`, `runNumber`, `reconstitutedModuleLabel`, `integratedSummary`, `verbosity`, @@ -213,6 +218,17 @@ for PBI — it remains available for any future workflow that needs per-index linear overrides on keys the target module actually accepts. +#### Update 2026-04-22: MDC2025aj accepts firstSubRun/firstEvent + +Offline PR #1799 + Production PR #533 (both merged 2026-04-15) added +`firstSubRunNumber` and `firstEventNumber` as optional `fhicl::Atom` +entries in `PBISequence_source.cc` with default 0. The MDC2025aj SimJob +backing (published 2026-04-22) ships this schema, so PBI jobs built +against `MDC2025aj/setup.sh` accept `event_id_per_index` overrides for +those keys. `source.maxEvents` is still rejected. `data/mdc2025/pbi_sequence.json` +uses this to assign globally-unique event numbers across indices +(verified: index 0 → 0, index 7 → 7000 at step=1000). + ### `mu2e -n ` injects maxEvents, which PBISequence rejects Passing `-n` on the `mu2e` command line causes art to inject @@ -248,6 +264,47 @@ automatically by `pbi_sequence.py`): } ``` +### MDC2025aj mu2e-trig-config schema drift (2026-04-23) + +MDC2025aj Musings ships a `mu2e-trig-config` package whose +`core/filters/trigCalFilters.fcl` uses an older `FilterEcalNNTrigger` +schema than the `backing/Offline/v13_08_00` C++ module expects. +Concretely, `trigCalFilters.fcl:56` sets: + +- `caloBkgMVA` (+ `caloBkgMVA.MVAWeights`) + +while the C++ `FilterEcalNNTrigger` Config now requires: + +- `caloMVACollection: art::InputTag` +- `minRtoTest`, `minTtoTest`, `maxEtoTest`, `maxRtoTest`, `maxTtoTest: float` + +Symptom: running any mix FCL on MDC2025aj that exercises the trigger +chain (all of `Production/JobConfig/mixing/Mix.fcl`'s flavors) aborts at +ModuleConstruction with: + +``` +Module label: CaloMVANNCEFilter +module_type : FilterEcalNNTrigger + Missing parameters: caloMVACollection, minRtoTest, … + Unsupported params: caloBkgMVA, caloBkgMVA.MVAWeights +``` + +**Not a prodtools issue.** Fix lives upstream — `mu2e-trig-config` +needs a refresh in the next Musings cut. Until then, mix jobs on aj +cannot run end-to-end in production. + +**Validated workaround for local sanity checks (2026-04-23):** source +`muse setup SimJob MDC2025ai` instead of aj and run the same aj-built +FCL. The #include paths resolve against ai's `mu2e-trig-config` (which +matches its own Offline v13_07_00 cleanly), RootInput reads the +aj-tagged dts file transparently, and `mu2e -c ... -n 1` completes +with exit 0 and a full TrigReport. Good enough to confirm overlay +wiring; do not use for production (wrong Offline version). + +Hit on 2026-04-23 while validating the stage-2 PBI mix overlay; jobdef +generation + fcldump overlay check were both clean, only the +`mu2e -c -n 1` schema validation tripped on it under aj env. + ### Use a recent enough campaign — MDC2025ac is stale Initial test against `MDC2025ac` hit two Offline-side blockers: @@ -278,6 +335,323 @@ more than a few months old, check whether a newer Musings (higher letter suffix on MDC20XX) has the fix before working around it locally. +## Stage 2: mixing PBI into dig + +The stage-1 outputs (`dts.mu2e.PBI_..art`) contain +`ProtonBunchIntensity` products and nothing else — no primary particles, +no detector steps. They are consumed as *input* to a Mix.fcl variant +that pulls PBI from the file instead of generating it inline. + +**Hook fcl:** `Production/JobConfig/mixing/NoPrimaryPBISequence.fcl`. +It includes `mixing/NoPrimary.fcl` (the standard mix-no-primary path) +and overrides `physics.producers.PBISim` with `NullProducer`, so the +per-event PBI values come from the source file's reconstituted +`ProtonBunchIntensity` stream rather than being regenerated. + +### mix.json entry + +**Cross-version configuration (2026-04-23):** inputs sourced from the +aj stage-1 production (`dts.mu2e.PBINormal_33344.MDC2025aj.art` — in +SAM with 26 files), but the mix step itself runs on MDC2025ai to +sidestep the aj trig-config drift (see Gotchas). PBI values in the +input files are just numbers — reading them under ai's Offline +v13_07_00 is transparent. + +```json +{ + "input_data": [ + {"dts.mu2e.PBINormal_33344.MDC2025aj.art": 1}, + {"dts.mu2e.PBIPathological_33344.MDC2025aj.art": 1} + ], + "dsconf": ["MDC2025ai_best_v1_3"], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh"], + "fcl": ["Production/JobConfig/mixing/Mix.fcl"], + "fcl_overrides": [{ + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_3", + "#include": "Production/JobConfig/mixing/NoPrimaryPBISequence.fcl", + "outputs.Output.fileName": "dig.mu2e.{desc}.{dsconf}.sequence.art" + }] +} +``` + +Same overlay pattern as the existing NoPrimary mix entries: `Mix.fcl` +as the base, the NoPrimary-flavored variant injected through +`fcl_overrides["#include"]`. Pileup datasets, `pbeam`, `merge_events` +match the sibling NoPrimary entries. + +Revert `dsconf`/`simjob_setup` to `MDC2025aj` once the `mu2e-trig-config` +package refreshes upstream and a post-fix aj Musings cut ships. + +### Output tier + +The mix job emits **dig-tier** directly (`dig.mu2e.*.art`), not dts — +matching the existing NoPrimary mix entries in `data/mdc2025/mix.json`. +No separate `digi.json` step is required for this chain; the Mix + +NoPrimary overlay wires in digitization. + +### dsconf / DB version + +`MDC2025aj_best_v1_3` was chosen by inheriting ag's most recent +version (`MDC2025ag_best_v1_3`) on the assumption calibrations carry +over to aj unchanged. **This requires `Sim_best / v1_3` to be +registered in DbService for aj** — jobs will fail at conditions lookup +if the version hasn't been cut. Verify before a production push. + +### `mixconf` + +Running counter across mix.json entries (0, 1, 2 for existing; +3 for the new PBI entry). Likely a random-seed / pileup-offset +discriminator ensuring independence across configs. + +### Validation status (2026-04-23) + +- Jobdefs build cleanly: `json2jobdef --json data/mdc2025/mix.json + --dsconf MDC2025aj_best_v1_3` produces 2 tarballs × 26 jobs each + (`cnf.mu2e.PBI{Normal,Pathological}_33344Mix1BB.MDC2025aj_best_v1_3.0.tar`). +- `fcldump --local-jobdef ... --index 0` confirms the overlay is wired + in the right order: `Mix.fcl` → `OneBB.fcl` → DB overrides → + `#include "Production/JobConfig/mixing/NoPrimaryPBISequence.fcl"`. + Source points at `dts.mu2e.PBINormal_33344.MDC2025aj.art` on tape; + pileup aux inputs resolve with correct multiplicities. +- **End-to-end `mu2e -c ... -n 1` on aj env** is blocked by an + unrelated upstream bug — see Gotchas → "MDC2025aj mu2e-trig-config + schema drift (2026-04-23)". +- **End-to-end validation on MDC2025ai env passes** (2026-04-23, exit 0, + full TrigReport). The same aj-built FCL run under `muse setup SimJob + MDC2025ai` completes mixing + trigger + digitization on 1 event. + RootInput reads the aj-tagged dts file transparently; the ai Offline + v13_07_00 doesn't exercise aj's PBISequence source schema, so the + cross-version mix validates cleanly. +- **mix.json entry switched to ai** (2026-04-23): pending the aj + trig-config fix, the production mix entry is + `MDC2025ai_best_v1_3` + `simjob_setup: MDC2025ai/setup.sh` with + aj inputs. `mu2e -c ... -n 1` on the resulting fcl produces + `dig.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.001430_00000001.art` + (1.57 MB, 1 event). +- **Production push completed (2026-04-24 UTC).** Via + `/mu2epro-run MDC2025ai json2jobdef --json data/mdc2025/mix.json + --dsconf MDC2025ai_best_v1_3 --prod --jobdefs + poms_map/MDC2025-025.json`. Both tarballs are SAM-declared: + - `cnf.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.0.tar` at + `/pnfs/.../phy-etc/cnf/mu2e/PBINormal_33344Mix1BB/MDC2025ai_best_v1_3/tar/…` + - `cnf.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.0.tar` at + `/pnfs/.../phy-etc/cnf/mu2e/PBIPathological_33344Mix1BB/MDC2025ai_best_v1_3/tar/09/e5/…` + + POMS map `MDC2025-025.json` extended in place (aj stage-1 × 52 + ai + mix × 52 = 104 jobs). SAM index `iMDC2025-025` deleted and recreated + with all 4 jobdef tarballs; POMS will pick up the 52 new mix jobs on + its next scan. + +- **POMS grid run completed (2026-04-24, ~1-hour turnaround).** Verified + via metacat MCP query on 2026-04-24: + - Tarballs declared 05:00 UTC → `dig.*` output datasets declared + 06:00 UTC (first file 05:12 UTC, last by 05:30 UTC for one variant). + - `dig.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.art`: **26 files**, + sizes ranging 0.87–1.92 GB per subrun (size tracks PBI intensity per + chunk — higher PBI = more detector activity = larger output). + - `dig.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.art`: **26 + files** (same structure). + - Sibling `log.*` datasets (26 files each) also registered. + All 52 mix jobs succeeded on the first POMS dispatch. + +- **`event_id_per_index` verified end-to-end in production.** Sample + `dig` file `...001430_00000021.art` from the PBINormal dataset has + metadata: + - `rs.first_subrun: 21`, `rs.last_subrun: 21` + - `rse.first_event: 21001`, `rse.last_event: 22000`, `rse.nevent: 1000` + These match the formula we set in + `data/mdc2025/pbi_sequence.json` — subrun offset=0 step=1 → + index 21 gives subrun 21; event offset=0 step=1000 → index 21 + gives events 21001..22000. Globally unique `(run, subrun, event)` + tuples confirmed across the dataset. The Offline PR #1799 + + Production #533 chain is delivering the intended behavior in + production, not just in dry-run. + +## Stage 3: reco of the PBI dig outputs + +Adds a `dig → mcs` step using `Production/JobConfig/recoMC/OnSpill.fcl`, +mirroring the ag-style entry pattern in `data/mdc2025/reco.json` (the +`MDC2025ag_best_v1_3` FlatGamma reco entry — non-Triggered dig inputs +through OnSpill). + +### reco.json entry + +```json +{ + "dsconf": ["MDC2025ai_best_v1_3"], + "tarball_append": "-reco", + "fcl": ["Production/JobConfig/recoMC/OnSpill.fcl"], + "input_data": [ + {"dig.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.art": 1}, + {"dig.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.art": 1} + ], + "fcl_overrides": [{ + "outputs.LoopHelixOutput.fileName": "mcs.owner.{desc}.version.sequencer.art", + "services.DbService.purpose": "Sim_best", + "services.DbService.version": "v1_3" + }], + "inloc": ["tape"], + "outloc": [{"*.art": "disk"}], + "simjob_setup": ["/cvmfs/mu2e.opensciencegrid.org/Musings/SimJob/MDC2025ai/setup.sh"] +} +``` + +Knob rationale: +- **`tarball_append: "-reco"`** — input dig and output mcs share + `desc` + `dsconf`, so the cnf tarball name would otherwise collide + with the dig stage's tarball. +- **`simjob_setup: MDC2025ai`** — same reason as Stage 2: aj is still + blocked by the `mu2e-trig-config` schema drift; reco stays on ai + matching the dig dsconf. +- **Merge factor 1** — dig files are 0.87–1.92 GB; one dig file + per reco job is the safe default. +- **`outloc: disk`** — matches existing recoMC entries; mcs goes to + tape only after QA. + +### DbService gotcha (2026-04-25) + +Without explicit `services.DbService.purpose: "Sim_best"` and +`services.DbService.version: "v1_3"` overrides, reco fails at +`ProtonBunchTimeFromStrawDigis` with: + +``` +DbHandle could not get TID (Table ID) from DbEngine for TrkPreampStraw +You are currently using DB calibration set EMPTY -1/-1/-1 +``` + +The existing `MDC2025af_best_v1_3` and `MDC2025ag_best_v1_3` reco +entries in `data/mdc2025/reco.json` **do not** override these — yet +they were the canonical pattern we built against. Two possibilities: + +1. The af/ag entries were never smoke-tested with `mu2e -c` + (only built as jobdefs), and would silently hit the same + `EMPTY -1/-1/-1` failure if run. +2. Some other code path (POMS env? grid wrapper? a `mu2eOps` + include?) injects `Sim_best` / `v` based on the `_best_v` + dsconf suffix, and only the local `mu2e -c` test misses it. + +Until that's resolved, **always set DbService overrides explicitly** +for new reco entries — derive purpose from the dsconf base +(`Sim_best`) and version from the suffix (`v1_3` for +`*_best_v1_3`). The mix.json entries already do this, so the +pattern is consistent. + +### Local smoke test + +```bash +# Generate jobdefs (under user account, no SAM push) +/mu2e-run MDC2025ai json2jobdef --json data/mdc2025/reco.json \ + --dsconf MDC2025ai_best_v1_3 + +# FCL for index 0 +/mu2e-run MDC2025ai jobfcl \ + --jobdef cnf..PBINormal_33344Mix1BB-reco.MDC2025ai_best_v1_3.0.tar \ + --index 0 --default-location tape --default-protocol root > test_reco.fcl + +# 1-event run (requires htgettoken for xrootd auth on personal shell) +/mu2e-run MDC2025ai mu2e -c test_reco.fcl -n 1 +``` + +### Validation status (2026-04-25) + +- Both jobdef tarballs build cleanly: 26 jobs each (one per dig file). +- 1-event smoke test on PBINormal index 0 (input file + `dig.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.001430_00000021.art`): + **exit 0**, all reco modules ran (KKDe/Dmu/Ue/Umu, helix finders, + calo, crv, makeSH/PH, LoopHelixOutput each Visited=1, Passed=1). + CPU 1.57s, VmPeak 1.98 GB. Output: + `mcs..PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.001430_00000021.art` + (568 KB for 1 event). +- Per-index sequencer carries through from dig to mcs cleanly: dig + index 21 → mcs `001430_00000021`, preserving the + `event_id_per_index` chain (subrun 21, events 21001..22000). +- PBIPathological smoke test (index 0, input file + `dig.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.001430_00000012.art`): + **exit 0**, same reco-module pattern as Normal, CPU 1.55s, VmPeak + 1.98 GB, output + `mcs..PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.001430_00000012.art`. + Both PBI flavors validated under MDC2025ai env. + +### Production push (2026-04-25 UTC) + +Pushed via — **note: target the existing PBI POMS map +(`MDC2025-025.json`), do not allocate a new map number**. The PBI +chain's stages 1+2 already live in `MDC2025-025`; reco extends that +map in place to keep the entire chain in one POMS scan target. + +```bash +/mu2epro-run MDC2025ai json2jobdef \ + --json data/mdc2025/reco.json \ + --dsconf MDC2025ai_best_v1_3 \ + --prod \ + --jobdefs /exp/mu2e/app/users/mu2epro/production_manager/poms_map/MDC2025-025.json +``` + +Both tarballs SAM-declared (verified via `samweb list-files`): + +- `cnf.mu2e.PBINormal_33344Mix1BB-reco.MDC2025ai_best_v1_3.0.tar` +- `cnf.mu2e.PBIPathological_33344Mix1BB-reco.MDC2025ai_best_v1_3.0.tar` + → at `/pnfs/mu2e/persistent/datasets/phy-etc/cnf/mu2e/PBIPathological_33344Mix1BB-reco/MDC2025ai_best_v1_3/tar/85/94/...` + +`MDC2025-025.json` extended in place from 4 → 6 entries (Stage 1 × 2 ++ Stage 2 × 2 + Stage 3 × 2). SAM index `iMDC2025-025` deleted and +recreated by `mkidxdef --prod`, now def_id 218087, dimension +`dh.dataset etc.mu2e.index.000.txt and dh.sequencer < 0000156` +covering all 156 jobs (52 per stage). POMS will pick up the 52 +new reco jobs on its next scan. + +#### Process note: extend, don't allocate + +First push attempt allocated a fresh `MDC2025-026.json` — +incorrect; the PBI chain belongs in one map. Remediation +(2026-04-25 ~11:53 UTC): + +1. Re-ran `json2jobdef ... --prod --jobdefs MDC2025-025.json` — + tarballs already in SAM so pushOutput no-oped (`already exists + on SAM, skipping push`); entries appended; `iMDC2025-025` + regenerated. +2. Removed orphan map file `MDC2025-026.json`. +3. Deleted orphan SAM index `iMDC2025-026` (the `samweb + delete-definition` CLI hit a `RecursionError` deep in + `urllib`/`socket` under one shell; succeeded under another with + no apparent state difference — quirk worth a wider note if it + recurs). + +The convention is now codified in the `/poms-push` skill — +auto-detects the right map by scanning existing entries for the +workflow family and prints the recommended `/mu2epro-run` +invocation before any push. + +#### Production grid run completed (2026-04-25 ~12:30 UTC) + +POMS dispatched and completed all 52 reco jobs within ~30 minutes +of the SAM index recreation. Verified via +`listNewDatasets --completeness`: + +``` + COUNT DATASET COMPLETENESS + ----- ------- ------------ + 26 mcs.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.art 26/26 + 26 mcs.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.art 26/26 +``` + +Sibling `log.*` datasets also landed (26 files each). Full PBI +chain (dts → dig → mcs) is now in production with globally-unique +`(run, subrun, event)` tuples preserved end-to-end. + +Expected outputs once POMS dispatches (mirroring the Stage 2 grid +turnaround of ~1 hour): + +- `mcs.mu2e.PBINormal_33344Mix1BB.MDC2025ai_best_v1_3.art` (26 files) +- `mcs.mu2e.PBIPathological_33344Mix1BB.MDC2025ai_best_v1_3.art` (26 files) +- sibling `log.*` datasets (26 files each) + +Sequencers will preserve the dig→mcs per-index mapping: e.g. dig +`001430_00000021` → mcs `001430_00000021` (run 1430, subrun 21, +events 21001..22000), keeping globally-unique `(run, subrun, event)` +tuples through the full chain. + ## Open questions - If PBI job event numbering needs to be globally unique across diff --git a/wiki/pages/poms-reference.md b/wiki/pages/poms-reference.md new file mode 100644 index 0000000..8c65e93 --- /dev/null +++ b/wiki/pages/poms-reference.md @@ -0,0 +1,188 @@ +--- +title: POMS reference (FNAL Production Operations Management System) +tags: [reference, infra, poms] +sources: [2026-04-28-poms-architecture, 2026-04-28-poms-sam-wiring, 2026-04-28-poms-user-ops] +updated: 2026-04-28 +--- + +# POMS reference + +Production Operations Management System — Fermilab's workflow orchestrator +above SAM, jobsub_lite, and art. Code: https://github.com/fermitools/poms. +Mu2e instance: https://poms.fnal.gov. + +This page synthesizes a 2026-04-28 research pass into POMS internals. +Verified claims are stated plainly; **unverified** items are flagged. +For the original raw research see the three `raw/` sources in +frontmatter. + +## What POMS is (and isn't) + +- **Is:** an orchestrator that wraps `jobsub_lite` (HTCondor submission), + `samweb` (data cataloging), and the `art` framework. Adds campaign + management, multi-stage dependencies, and automated recovery. +- **Is not:** SAM (file metadata + queries), jobsub_lite (low-level + submission), or art (event processing framework). + +## Data model + +``` +Campaign ──┬── CampaignStage ──┬── Submission ──┬── Job + │ │ ├── Job + │ │ └── Job + │ ├── Submission ──── ... + │ ├── JobType (template) + │ ├── LoginSetup (auth) + │ └── (SAM dataset name, dispatch params) + │ + └── CampaignStage ──── ... +``` + +| Entity | What it carries | Where in repo | +|---|---|---| +| **Campaign** | Top-level workflow container; experiment, default params | `webservice/CampaignsPOMS.py` | +| **CampaignStage** | One processing step. JobType ref, LoginSetup ref, **SAM dataset name** (the def POMS iterates), split strategy, completion criteria | `webservice/StagesPOMS.py` | +| **Submission** | A single launch of a stage. Tracks N jobs from creation → Located/Failed | `webservice/SubmissionsPOMS.py` | +| **JobType** | Reusable execution template (script, params, recovery cfg). Snapshotted at submission time for reproducibility | `webservice/MiscPOMS.py` | +| **LoginSetup** | Auth + launch host config | `webservice/MiscPOMS.py` | +| **DataDispatcherSubmission** | Bridge to Data Dispatcher when a stage uses DD instead of SAM | `webservice/DMRService.py` | + +Schema: `ddl/poms_ddl.sql`. ORM: `webservice/poms_model.py`. + +## Dispatch lifecycle + +1. **User triggers a launch** — Web UI button, `poms_client` call, or + POMS-internal cron. +2. **Submission created** — POMS writes a row tied to the CampaignStage + with snapshotted JobType + parameters; status `New`. +3. **Jobs submitted** — `poms_jobsub_wrapper` calls `jobsub_lite` for + each file in the stage's SAM def. One Condor job per file. +4. **Worker runs** — Worker gets `fname` env var = the assigned SAM + file. Runs the JobType's executable (e.g., `runmu2e --jobdesc `). +5. **POMS polls** — `submission_agent` daemon hits LENS every ~120s, + updates job/file status. +6. **Completion** — `wrapup_tasks` flips submission to `Located` (all + files delivered) or `Failed`. +7. **Recovery / cascade** — Configured recoveries re-dispatch failures. + Once Located, downstream stages auto-launch. + +## Mu2e-specific conventions + +The Mu2e instance runs the upstream POMS code but layers conventions: + +### Dropbox path + +POMS scans `/exp/mu2e/app/users/mu2epro/production_manager/poms_map/` +for `*.json` map files. Each map describes the jobs to dispatch: +tarball name, `njobs`, `inloc`, `outputs`, plus runner-specific +fields like `runner: "g4bl"`. See [pbi-sequence-workflow](pbi-sequence-workflow) +for an art-side example. + +### `i` SAM-def naming + +The Mu2e tool **`mkidxdef --prod`** (in `prodtools/bin/mkidxdef`) +creates a SAM def named `i` from a map file's stem. e.g., +`MDC2025-025.json` → `iMDC2025-025`. This is **a Mu2e tooling +convention, not a POMS hardcode.** POMS itself takes whatever SAM +def name is configured on the CampaignStage. + +The `iMDC2025-NNN` SAM defs are **content-agnostic placeholders** — +they contain `etc.mu2e.index.000..txt` files used solely for +job-count iteration (POMS dispatches one worker per file). They're +not tied to any campaign by content; **`iMDC2025-025` can be reused +across stages** that just need N parallel dispatches. Confirmed +by user 2026-04-28. + +### Dispatch decoupling possibility (unverified) + +A POMS stage's SAM-def field is configurable per-stage (via web UI +or `poms_client.update_campaign_stage()`). In principle, a stage +can dispatch `G4BL-000.json` against `iMDC2025-025` without renaming +either. **Not yet verified end-to-end** — the exact column name on +`campaign_stages` and the precise `update_campaign_stage()` payload +weren't pinned down by the 2026-04-28 research. + +## User operations + +### Web UI + +https://poms.fnal.gov · OIDC auth (no Kerberos required for UI). Top +navigation: Campaigns → click campaign → stages list → "Launch". + +### `poms_client` Python library + +Available somewhere under POMS distribution (UPS or `pip install`). +Key methods: + +```python +from poms_client.poms_client import pomsclient + +pc = pomsclient(experiment='mu2e') + +# List +pc.show_campaigns(experiment='mu2e') + +# Submit / launch +pc.get_submission_id_for(campaign_stage_id=18, input_dataset='...') +pc.launch_campaign_stage_jobs(campaign_name='X', stage_name='Y', limit=1000) + +# Inspect +pc.submission_details(submission_id=123) + +# Modify stage (changes campaign_stages row) +pc.update_campaign_stage(...) + +# Upload a campaign config (.ini) or a map file +pc.upload_wf('mycampaign.ini') +pc.upload_file('mymap.json') +``` + +### Standard Mu2e workflow (for art jobs) + +1. Build tarball + push to SAM: `json2jobdef --prod --jobdefs MDC2025-NNN.json` +2. Create SAM index: `mkidxdef --prod --jobdefs MDC2025-NNN.json` → + creates `iMDC2025-NNN` +3. Drop the map at `/exp/mu2e/app/users/mu2epro/production_manager/poms_map/MDC2025-NNN.json` +4. POMS auto-discovers (web UI shows it in the campaign's stages), + user clicks Launch, or POMS-side cron dispatches + +### Monitoring + +- **Web UI** — campaign/stage/submission pages with completion %, + file statuses, Metacat lineage links +- **FIFEmon** — jobsub-level logs (jobsub job id → live status) +- **Kibana** — detailed worker logs at FNAL ELK stack + +## Common pitfalls + +| Symptom | Cause | +|---|---| +| Map dropped, never dispatched | POMS stage not configured / not pointing at the right SAM def / dropbox file pattern not matching stage cfg | +| Workers all fail at SAM push | `mu2epro` token missing on worker; `pushOutput` can't auth — see [feedback_never_get_mu2epro_token](../../memory/feedback_never_get_mu2epro_token.md) | +| Stage dispatches N workers but 0 outputs | SAM dataset has 0 matching files; check `samweb count-files defname:i` | +| `samweb create-definition` redirect-loop | Token-auth path issue; **never** fall back to `voms-proxy-init` (Mu2e migrated to bearer tokens) — see `feedback_no_voms_proxy_init` | + +## Open questions (need verification on the running instance) + +1. **Exact column on `campaign_stages`** holding the SAM-def name + (likely `dataset` or `cs_dataset` — check via `\d campaign_stages` + on the POMS DB). +2. **Whether map filename auto-derives the stage's SAM def** by some + POMS convention, or only via Mu2e tooling glue. +3. **`poms_client.update_campaign_stage()` exact payload** for + changing the SAM def of an existing stage. +4. **Whether the running Mu2e POMS instance permits stages to use + shared/reused SAM defs** without admin intervention. + +## Pointers + +- Repo: https://github.com/fermitools/poms +- Mu2e POMS instance: https://poms.fnal.gov +- Mu2e wiki: https://mu2ewiki.fnal.gov/wiki/POMS +- Email: poms_announce@fnal.gov + +## Related local pages + +- [[pbi-sequence-workflow]] — concrete example of `MDC2025-025.json` map + iMDC2025-025 + Stage 3 reco dispatch +- `poms/main.cfg`, `poms/prolog.cfg`, `poms/fermigrid.cfg` in this repo — Mu2e-specific POMS submit configs (a `poms/g4bl.cfg` existed briefly to set an SL7 outer container; retired 2026-04-28 once g4bl gained a native AL9 spack build) +- `bin/mkidxdef` + `utils/mkidxdef.py` — the Mu2e i tool diff --git a/wiki/raw/2026-04-24-mu2e-aitools-skills.md b/wiki/raw/2026-04-24-mu2e-aitools-skills.md new file mode 100644 index 0000000..81d9bbc --- /dev/null +++ b/wiki/raw/2026-04-24-mu2e-aitools-skills.md @@ -0,0 +1,100 @@ +--- +title: Mu2e/aitools skills and MCP READMEs (source snapshot) +source_type: github-repo +source_url: https://github.com/Mu2e/aitools +fetched: 2026-04-24 +--- + +# Mu2e/aitools — source snapshot + +Snapshot of the upstream AI-tools repo at fetch time. Not refetched +automatically; treat as a point-in-time record. Synthesized page: +[[metacat-reference]]. + +## Repo shape + +``` +aitools/ +├── ai-instructions.md # Offline-focused entry point (build system, muse, spack) +├── skills/ +│ ├── building-offline-software/SKILL.md +│ ├── building-with-muse/SKILL.md +│ ├── building-with-smack/SKILL.md +│ ├── understanding-data-handling/SKILL.md +│ ├── finding-data-sam/SKILL.md +│ ├── finding-data-metacat/SKILL.md ← pulled +│ ├── coding-with-metacat/SKILL.md ← pulled +│ └── coding-with-query-engine/SKILL.md +└── mcp/ + ├── metacat/ ← pulled (README) + ├── sim-epochs/ ← pulled (README) + ├── dqm/ + └── code-index/ +``` + +Status at fetch time: 22 commits, 0 stars, 4 forks, Apache 2.0, +Python 70% / Shell 30%. + +## `finding-data-metacat/SKILL.md` — what we pulled + +- Environment setup (`mu2einit`, `muse setup ops`, Kerberos for auth). +- Full `metacat` CLI command table: `auth`, `dataset`, `file`, + `namespace`, `category`, `named_query`, `query`, `version`, + `validate`. +- Full `mdh` command table: `compute-crc`, `print-url`, `query-dcache`, + `create-metadata`, `declare-file`, `locate-dataset`, `copy-file`, + `prestage-files`, `verify-dataset`, `delete-files`, `upload-grid`. +- MQL patterns: `files from`, `where`, `order by`, `limit`, `offset`. +- Namespace & file naming (six-field DID convention). +- dCache storage tiers (`tape` / `disk` / `scratch`) and path prefixes. + +## `coding-with-metacat/SKILL.md` — what we pulled + +- Python client init (`MetaCatClient()`, env-driven vs explicit URL+token). +- Method reference table: read-only (`list_datasets`, `get_dataset`, + `get_dataset_files`, `get_file`, `query`) vs write + (`declare_file`, `create_dataset`, `update_dataset`, `move_files`, + `delete_file`, `retire_file`, etc.). +- MQL in Python: `client.query()` is a lazy generator; force with + `list(...)`. +- Scalable patterns: filter-then-count (not `with_counts=True` globally), + pagination via `limit/offset`. +- Concrete snippets: list datasets, dataset statistics, recent files + by timestamp, metadata completeness validation, URL generation from + metacat + `mdh print-url`, art-job input list generation. +- `SafeMetaCat` wrapper class (blocks write methods unless + `ALLOW_WRITES=True`). +- Safety rules (read-only default, confirm before writes, don't hardcode + server URLs, don't use SQL syntax, force query evaluation). + +## `mcp/metacat/README.md` — what we pulled + +- Install: `python3 -m venv .venv; pip install -U pip -r requirements.txt -e .` +- Tools: `discover_datasets`, `get_dataset_details`, + `query_dataset_files`, `get_server_info` (all read-only). +- Claude Code registration via `mcpServers.metacat-readonly.command` + pointing at `scripts/start_mcp.sh`. +- Auth: env-provided via `MetaCatClient()` — no MCP-layer auth. +- Runs `setupmu2e-art.sh` + `muse setup ops` at startup. +- Group deployment via install script with version tagging. + +## `mcp/sim-epochs/README.md` — what we pulled + +- Stdio MCP over `data/sim_catalog.json` (overridable via + `SIM_EPOCHS_FILE`). +- Tools: `get_simulation_epochs()`, `get_datasets_for_epoch(epoch)`. +- Catalog format (preferred): `{"epochs": [{"name": "MDC2025ad", + "datasets": [...]}]}`. Compact: `{"MDC2025ad": [...], ...}`. +- Install pattern same as metacat. Env controls: `MCP_PYTHON`, + `MCP_BASH_SETUP`, `MCP_PYTHONPATH_MODE`. +- Shared deployment convention: `/shared/mcp/sim-epochs/releases/0.1.0/` + + `current` symlink + centralized registry. + +## Not pulled + +- `finding-data-sam/SKILL.md` — prodtools already runs on SAM; we have + internal knowledge. +- `coding-with-query-engine/SKILL.md` — DBReader/DbIdList not a + prodtools concern. +- `building-*/SKILL.md` — not a prodtools concern. +- `mcp/dqm`, `mcp/code-index` — not currently used. diff --git a/wiki/raw/2026-04-27-g4bl-runner-integration.md b/wiki/raw/2026-04-27-g4bl-runner-integration.md new file mode 100644 index 0000000..043dfd2 --- /dev/null +++ b/wiki/raw/2026-04-27-g4bl-runner-integration.md @@ -0,0 +1,38 @@ +# in-session 2026-04-27: G4Beamline runner integration into prodtools + +What was added: +- New `runner: "g4bl"` field in jobdesc/JSON schema (sibling to existing 'template' / 'direct_input' / normal modes), routed through `validate_jobdesc` returning 'g4bl' mode. +- New function `process_g4bl_jobdef(jobdesc_entry, fname, args)` in `utils/prod_utils.py` that executes g4bl in-place (no separate `mu2e -c` step). Writes a transient bash script to the output dir; runs `apptainer exec` against an SL7 container; binds /cvmfs, /tmp, embed_dir, $HOME; runs `source setupmu2e-art.sh; setup G4beamline; cd embed_dir; g4bl Num_Events=N First_Event=F param histoFile=.root`. +- New `runmu2e.py` g4bl branch that calls `process_g4bl_jobdef` and skips the FCL/mu2e dispatch. +- New constant `DEFAULT_G4BL_CONTAINER = /cvmfs/singularity.opensciencegrid.org/fermilab/fnal-dev-sl7:latest` (overridable via JSON `container` field). +- New `poms/g4bl.cfg` mirroring `fermigrid.cfg` but with `+SingularityImage` pointing at the SL7 dev image — for grid submission, the outer container *is* SL7, so no nesting needed at runtime. +- JSON schema fields: `runner`, `embed_dir` (single root dir to bundle), `main_input` (filename inside embed_dir), `events_per_job`, `njobs`. No `fcl`, no `simjob_setup` (different mechanism). +- Files: data/mdc2025/g4bl.json (canonical production config), test/g4bl_smoke_jobdesc.json (smoke fixture). + +Decisions made (with rationale, ranked by surprise): + +1. Embed strategy = "single root dir + main_input filename" (NOT separate input_lattice + embed_paths list). Mirrors what the user already does manually (`cd $IN_DIR && g4bl $(basename $IN)`). Geometry/*.txt files reference each other by relative path so they MUST travel with the .in. cvmfs-absolute references (BField maps under /cvmfs/mu2e.opensciencegrid.org/DataFiles/BFieldMaps/GA05/) stay where they are — container has /cvmfs bound. Default `embed_exclude` drops `*.root`/`*.tar`/`root_archive_*` so 1.7M of past outputs in the source dir don't bloat the tarball; embedded footprint for Mu2E.in is ~560K. + +2. First_Event indexing is 0-based linear: `first_event = job_index * events_per_job + 1`. Matches the existing prodtools `range(njobs)` dispatch convention. NOT seed-based (defer if needed later). + +3. Container path NOT in JSON — code constant + POMS cfg. Two homes: + - Local-runner fallback: `DEFAULT_G4BL_CONTAINER` constant in prod_utils.py + - Grid: `+SingularityImage` in poms/g4bl.cfg (outer container set by Condor) + Single source of truth as a string in code; cfg duplicates it but cfg files don't reference Python, so the duplication is acceptable. JSON's `container` field is opt-in override. + +4. fnal-wn-sl7 doesn't exist — only fnal-dev-sl7. Confirmed via /cvmfs/singularity.opensciencegrid.org/fermilab/ listing. fnal-dev-sl7 works as a Condor `+SingularityImage` (Condor accepts any path; the wn-* naming is FNAL convention, not requirement). Avoids nested containers entirely on grid. + +5. Reuse cnf.*.tar prefix (NOT g4bl.cnf.*.tar) — uniform tarball naming across runners; content differs but naming convention stays. + +Two non-obvious gotchas (would have been missed without smoke test): + +A) **`--cleanenv` is mandatory** when launching apptainer from a Python subprocess whose parent has AL9 mu2e env sourced. The AL9 PYTHONHOME / UPS_DIR / PRODUCTS env vars leak into the SL7 container and break setupmu2e-art.sh's UPS init silently. Symptom: `bash: setup: command not found` at line 3 of the runner script, despite `source` returning 0. The same command run via shell `eval` works without --cleanenv (bash doesn't inject those vars). Discovered after multiple incorrect bisections (set -e? && vs newline? bash -lc vs bash