Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split dqm configuration tests in smaller parts to avoid timeouts #34306

Merged
merged 1 commit into from Jul 2, 2021
Merged

Split dqm configuration tests in smaller parts to avoid timeouts #34306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 2 additions & 0 deletions DQMOffline/Configuration/scripts/cmsswSequenceInfo.py
View file
Open in desktop
Expand Up @@ -496,6 +496,7 @@ def do_GET(self):
parser.add_argument("--steps", default="ALCA,ALCAPRODUCER,ALCAHARVEST,DQM,HARVESTING,VALIDATION", help="Which workflow steps to inspect from runTheMatrix.")
parser.add_argument("--sqlite", default=False, action="store_true", help="Write information to SQLite DB instead of stdout.")
parser.add_argument("--dbfile", default="sequences.db", help="Name of the DB file to use.")
parser.add_argument("--infile", default=INFILE, help="LFN/PFN of input file to use. Default is %s" % INFILE)
parser.add_argument("--threads", default=None, type=int, help="Use a fixed number of threads (default is #cores).")
parser.add_argument("--limit", default=None, type=int, help="Process only this many sequences.")
parser.add_argument("--offset", default=None, type=int, help="Process sequences starting from this index. Used with --limit to divide the work into jobs.")
Expand All @@ -514,6 +515,7 @@ def do_GET(self):
tp = ThreadPool(args.threads)
stp = ThreadPool(args.threads)

INFILE = args.infile
if args.serve:
serve()
elif args.workflow or args.runTheMatrix:
Expand Down
30 changes: 13 additions & 17 deletions DQMOffline/Configuration/test/BuildFile.xml
View file
Open in desktop
@@ -1,19 +1,15 @@
<!-- To make the tests run in parallel, we chunk up the work into arbitrary sets of 20 sequences. -->
<test name="TestDQMOfflineConfiguration0" command="runtests.sh 20 0"/>
<test name="TestDQMOfflineConfiguration20" command="runtests.sh 20 20"/>
<test name="TestDQMOfflineConfiguration40" command="runtests.sh 20 40"/>
<test name="TestDQMOfflineConfiguration60" command="runtests.sh 20 60"/>
<test name="TestDQMOfflineConfiguration80" command="runtests.sh 20 80"/>
<test name="TestDQMOfflineConfiguration100" command="runtests.sh 20 100"/>
<test name="TestDQMOfflineConfiguration120" command="runtests.sh 20 120"/>
<test name="TestDQMOfflineConfiguration140" command="runtests.sh 20 140"/>
<test name="TestDQMOfflineConfiguration160" command="runtests.sh 20 160"/>
<test name="TestDQMOfflineConfiguration180" command="runtests.sh 20 180"/>
<test name="TestDQMOfflineConfiguration200" command="runtests.sh 20 200"/>
<test name="TestDQMOfflineConfiguration220" command="runtests.sh 20 220"/>
<test name="TestDQMOfflineConfiguration240" command="runtests.sh 20 240"/>
<test name="TestDQMOfflineConfiguration260" command="runtests.sh 20 260"/>
<test name="TestDQMOfflineConfiguration280" command="runtests.sh 20 280"/>
<!-- Download input file locally so that all tests can use it -->
<set name="INFILE_NAME" value="004D960A-EA4C-E811-A908-FA163ED1F481.root"/>
<set name="INFILE" value="/store/data/Run2018A/EGamma/RAW/v1/000/315/489/00000/${INFILE_NAME}"/>
<test name="GetTestDQMOfflineConfigurationFile" command="edmCopyUtil ${INFILE} $(LOCALTOP)/tmp/"/>

<!-- To make the tests run in parallel, we chunk up the work into arbitrary sets of 10 sequences. -->
<test name="TestDQMOfflineConfiguration" command="runtests.sh ${step_value} ${value} file://${LOCALTOP}/tmp/${INFILE_NAME}" for="0,290,10">
<flags PRE_TEST="GetTestDQMOfflineConfigurationFile"/>
</test>

<!-- To make sure we actually got all sequences, the last check checks that there are no sequences beyond the last test -->
<!-- This might need to updated when the number of distinct sequences grows, add more rows above and change the number here. -->
<test name="TestDQMOfflineConfigurationGotAll" command="cmsswSequenceInfo.py --runTheMatrix --steps DQM,VALIDATION --limit 50 --offset 300 --threads 1 | grep 'Analyzing 0 seqs'"/>
<test name="TestDQMOfflineConfigurationGotAll" command="cmsswSequenceInfo.py --runTheMatrix --steps DQM,VALIDATION --infile file://${LOCALTOP}/tmp/${INFILE_NAME} --limit 50 --offset 300 --threads 1 | grep 'Analyzing 0 seqs'">
<flags PRE_TEST="GetTestDQMOfflineConfigurationFile"/>
</test>
7 changes: 6 additions & 1 deletion DQMOffline/Configuration/test/runtests.sh
View file
Open in desktop
Expand Up @@ -15,7 +15,12 @@ if [[ -n $1 && -n $2 ]]; then
THREADS="--threads 1"
fi

cmsswSequenceInfo.py --runTheMatrix --steps DQM,VALIDATION $SECTION --dbfile "$DBFILE" $THREADS
INFILE=""
if [[ -n $3 ]] ; then
INFILE="--infile $3"
fi

cmsswSequenceInfo.py --runTheMatrix --steps DQM,VALIDATION $INFILE $SECTION --dbfile "$DBFILE" $THREADS
sqlite3 "$DBFILE" > "legacymodules-${DBFILE}.txt" <<SQL
SELECT edmfamily, edmbase, classname, instancename, step, seqname, wfid
FROM plugin
Expand Down