Two relvals failing in Geometry #32181

mrodozov · 2020-11-19T12:05:17Z

Dears,

in the last night IB there are two dd4hep failures
https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_11_2/2020-11-18-2300?selectedArchs=slc7_amd64_gcc820&selectedFlavors=X&selectedStatus=failed

the relvals failing were added in:
#32096

mrodozov · 2020-11-19T12:05:25Z

assign geometry

cmsbuild · 2020-11-19T12:05:35Z

New categories assigned: geometry

@Dr15Jones,@cvuosalo,@mdhildreth,@makortel,@ianna,@civanch you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild · 2020-11-19T12:05:37Z

A new Issue was created by @mrodozov Mircho Rodozov.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel · 2020-11-19T14:25:28Z

The error (from 11624.911) is

PersistencyIO    INFO  +++ Set Streamer to dd4hep::OpaqueDataBlock
CompactLoader    INFO  +++ Processing compact file: /cvmfs/cms-ib.cern.ch/nweek-02655/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-11-18-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2021.xml with flag (null)
DD4CMS           INFO  +++ Processing the CMS detector description file:///cvmfs/cms-ib.cern.ch/nweek-02655/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-11-18-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2021.xml
Detector         INFO  *********** Created World volume with size: 10100 10100 45000
Detector         INFO  +++ Patching names of anonymous shapes....
DDDefinition     INFO  +++ Finished processing file:///cvmfs/cms-ib.cern.ch/nweek-02655/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-11-18-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2021.xml
----- Begin Fatal Exception 19-Nov-2020 08:44:52 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Running EventSetup component GEMGeometryESModule/'gemGeometry
Exception Message:
A std::exception was thrown.
dd4hep:  : value=384 [Evaluation error]
----- End Fatal Exception -------------------------------------------------

civanch · 2020-11-19T16:52:34Z

@cvuosalo , @slomeo , what is different for GEM from your local runs?

silviodonato · 2020-11-19T17:00:41Z

Perhaps the error in ASAN might be helpful https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_2_ASAN_X_2020-11-18-2300/pyRelValMatrixLogs/run/11624.911_TTbar_13+2021_DD4hep+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA/step1_TTbar_13+2021_DD4hep+TTbar_13TeV_TuneCUETP8M1_GenSim+Digi+Reco+HARVEST+ALCA.log#/

%MSG-i ThreadStreamSetup:  (NoModuleName) 19-Nov-2020 09:07:30 CET pre-events
setting # threads 4
setting # streams 4
%MSG
WARNING: MCParticlePairFilter : size of some vectors not matching with 2!!
PersistencyIO    INFO  +++ Set Streamer to dd4hep::OpaqueDataBlock
CompactLoader    INFO  +++ Processing compact file: /cvmfs/cms-ib.cern.ch/nweek-02655/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-18-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2021.xml with flag (null)
PluginService    ERROR Factory requested: DDDefinition_XML_reader (N10__cxxabiv120__function_type_infoE) :bad any_cast
PluginService    ERROR Stub is invalid!
----- Begin Fatal Exception 19-Nov-2020 09:08:26 CET-----------------------
An exception of category 'StdException' occurred while
   [0] Processing global begin Run run: 1
   [1] Calling method for module OscarMTProducer/'g4SimHits'
   [2] Using EventSetup component DDCompactViewESProducer/'' to make data DDCompactView/'' in record IdealGeometryRecord
   [3] Running EventSetup component DDDetectorESProducer/'
Exception Message:
A std::exception was thrown.
dd4hep: Failed to locate plugin to interprete files of type "DDDefinition" - no factory:DDDefinition_XML_reader. 		No factory with name Create(DDDefinition_XML_reader) for type DDDefinition_XML_reader found.
		Please check library load path and/or plugin factory name.
dd4hep: while parsing /cvmfs/cms-ib.cern.ch/nweek-02655/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_ASAN_X_2020-11-18-2300/src/Geometry/CMSCommonData/data/dd4hep/cmsExtendedGeometry2021.xml
dd4hep: with plugin:DD4hep_CompactLoader
----- End Fatal Exception -------------------------------------------------

silviodonato · 2020-11-19T18:50:01Z

Meanwhile I've found something weird

cmsDriver.py TTbar_13TeV_TuneCUETP8M1_cfi  --conditions auto:phase1_2021_realistic -n 10 --era Run3 --eventcontent FEVTDEBUG --procModifiers dd4hep --relval 9000,50 -s GEN,SIM --datatier GEN-SIM --beamspot Run3RoundOptics25ns13TeVLowSigmaZ --geometry DD4hepExtended2021 --fileout file:step1.root  --no_exec

edmConfigDump  TTbar_13TeV_TuneCUETP8M1_cfi_GEN_SIM.py > dump.py

[sdonato@cmsanalysis DD4HEP]$ cat dump.py  | grep DDD
    fromDDD = cms.bool(False),
    DDDetector = cms.ESInputTag("",""),
    fromDDD = cms.bool(False),
    fromDDD = cms.untracked.bool(False)
    fromDDD = cms.bool(False)
process.hcalDDDRecConstants = cms.ESProducer("HcalDDDRecConstantsESModule",
process.hcalDDDSimConstants = cms.ESProducer("HcalDDDSimConstantsESModule",
    fromDDD = cms.bool(False),
    DDDetector = cms.ESInputTag("",""),
    fromDDD = cms.bool(False),
    fromDDD = cms.bool(True)
    fromDDD = cms.bool(True)
    fromDDD = cms.bool(False)
process.DDDetectorESProducer = cms.ESSource("DDDetectorESProducer",

fromDDD = cms.bool(True) looks suspicious

cvuosalo · 2020-11-19T19:25:53Z

In CMSSW_11_2_X_2020-11-18-2300, I ran:

runTheMatrix.py -l 11624.911,11642.911 --ibeos

They run all steps successfully and do not show the errors mentioned in this issue.
What is the difference between my test and an IB test?

cvuosalo · 2020-11-19T19:31:33Z

Note that in the IB, 11624.911 shows this error:

 [0] Running EventSetup component GEMGeometryESModule/'gemGeometry
Exception Message:
A std::exception was thrown.
dd4hep:  : value=384 [Evaluation error]

while 11642.911 shows this one:

   [1] Calling method for module SiStripFEDMonitorPlugin/'siStripFEDMonitor'
   [2] Using EventSetup component TkDetMapESProducer/'' to make data TkDetMap/'' in record TrackerTopologyRcd
   [3] Running EventSetup component TrackerGeometricDetESModule/'trackerNumberingGeometry
Exception Message:
A std::exception was thrown.
dd4hep: Evaluator : systax error : value=0.10349 [Evaluation error]

Could they be spurious? Could we really have two different real errors that somehow do not show up in the PR tests or direct runTheMatrix tests?

makortel · 2020-11-19T19:34:07Z

One clear difference between PR and IB tests is that PR tests are single-thread and IB tests are multi-threaded.

silviodonato · 2020-11-19T19:38:33Z

Right, the PR tests are ok indeed #31220 (comment)

silviodonato · 2020-11-19T19:40:22Z

The command used in the IB test is runTheMatrix.py -l limited -i all --job-reports -t 4 --ibeos

silviodonato · 2020-11-19T20:28:31Z

@cvuosalo @bsunanda is it ok this line ? It consumes DDDetector if fromDD4hep_ and we have DDDetector = cms.ESInputTag("","")
https://github.com/cms-sw/cmssw/blob/master/Geometry/DTGeometryBuilder/plugins/DTGeometryESModule.cc#L111

process.DTGeometryESModule = cms.ESProducer("DTGeometryESModule",
    DDDetector = cms.ESInputTag("",""),
    alignmentsLabel = cms.string(''),
    appendToDataLabel = cms.string(''),
    applyAlignment = cms.bool(True),
    attribute = cms.string('MuStructure'),
    fromDD4hep = cms.bool(True),
    fromDDD = cms.bool(False),
    value = cms.string('MuonBarrelDT')
)

process.idealForDigiDTGeometry = cms.ESProducer("DTGeometryESModule",
    DDDetector = cms.ESInputTag("",""),
    alignmentsLabel = cms.string('fakeForIdeal'),
    appendToDataLabel = cms.string('idealForDigi'),
    applyAlignment = cms.bool(False),
    attribute = cms.string('MuStructure'),
    fromDD4hep = cms.bool(True),
    fromDDD = cms.bool(False),
    value = cms.string('MuonBarrelDT')
)

cvuosalo · 2020-11-19T21:53:07Z

I ran the workflows with "-t 4" to allow multi-threading. One ran to completion successfully, and one crashed in step3, with a different message than before.
Multi-threading seems to induce semi-random behavior, at least in step3. Somehow the threads are occasionally stomping on each other. How can this problem be debugged?

fabiocos · 2020-11-19T21:57:50Z

@cvuosalo I confirm, in single threaded mode the workflow runs smoothly, in multi-thread I got a failure in step2 due to the evaluation of a constant in dd4hep. It looks like the issue depends on the memory access of a single job, this would explain why it does not seem to be systematically reproducible.

cvuosalo · 2020-11-19T22:38:22Z

Can we run these two workflows in the IB tests in single-threaded mode for now?

silviodonato · 2020-11-20T07:55:22Z

Can we run these two workflows in the IB tests in single-threaded mode for now?

The PR tests run single thread, so we are checking DD4HEP in each PR and this does not create any problem.
About the IB tests I think it is good to keep it multi-threaded even if it is crashing right now

makortel · 2020-11-23T22:36:22Z

#32249 suggests a culprit for the failures with multiple threads

makortel · 2020-11-25T13:51:06Z

The DD4Hep workflows seem also to generate large number of differences (possibly randomly) in PR tests, see e.g. #32270 (comment). Should we consider removing them from the PR tests until they become more stable?

bsunanda · 2020-11-25T14:06:16Z

Please do that. DD4Hep workflows are still being debugged. They should not be in the comparison lists.

…

________________________________ From: Matti Kortelainen [notifications@github.com] Sent: 25 November 2020 14:51 To: cms-sw/cmssw Cc: Sunanda Banerjee; Mention Subject: Re: [cms-sw/cmssw] Two relvals failing in Geometry (#32181) The DD4Hep workflows seem also to generate large number of differences (possibly randomly) in PR tests, see e.g. #32270 (comment)<#32270 (comment)>. Should we consider removing them from the PR tests until they become more stable? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#32181 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABGMZOUXZPKX4FHGGNA6GOTSRUDVTANCNFSM4T3JZUIQ>.

makortel · 2020-11-25T22:55:19Z

The ASAN failure in #32181 (comment) reproduces when run with single thread. Maybe that would be a good starting point for further debugging? (also #32181 (comment) and #32181 (comment))

silviodonato · 2020-12-04T08:53:15Z

solved by #32371
please note #32249 is still open

cvuosalo · 2020-12-09T18:00:23Z

In CMSSW_11_3_X_2020-12-08-1100, I have confirmed that workflows 11624.911 and 11642.911 run successfully to completion with 1000 events in both single- and multi-threaded mode.

dan131riley · 2021-01-19T20:32:46Z

Perhaps the error in ASAN might be helpful

I think the ASAN problem comes from the dd4hep plugin manager trying to use the type of the DDDefinition_XML_reader plugin before the plugin has been loaded. It is unrelated to the other exception. Unfortunately, I don't see an obvious way to fix it.

cmsbuild added geometry-pending pending-signatures labels Nov 19, 2020

silviodonato mentioned this issue Nov 19, 2020

Add DD4hep Run 3 workflows 11624.911 and 11642.91 to PR/IB tests #32096

Merged

makortel mentioned this issue Nov 23, 2020

DD4Hep uses a non-const global evaluator #32249

Closed

qliphy mentioned this issue Nov 24, 2020

Add low-pT electrons to MINIAOD, update ID, improve end user experience #31220

Merged

silviodonato closed this as completed Dec 4, 2020

silviodonato mentioned this issue Dec 4, 2020

Readd 1624.911, #2021 DD4hep from short matrix #32385

Merged

mrodozov mentioned this issue Jan 20, 2021

Update DD4Hep to latest on master cms-sw/cmsdist#6554

Merged

mrodozov mentioned this issue Jan 28, 2021

Update DD4Hep to latest on master cms-sw/cmsdist#6594

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Two relvals failing in Geometry #32181

Two relvals failing in Geometry #32181

mrodozov commented Nov 19, 2020

mrodozov commented Nov 19, 2020

cmsbuild commented Nov 19, 2020

cmsbuild commented Nov 19, 2020

makortel commented Nov 19, 2020

civanch commented Nov 19, 2020

silviodonato commented Nov 19, 2020

silviodonato commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

makortel commented Nov 19, 2020

silviodonato commented Nov 19, 2020

silviodonato commented Nov 19, 2020

silviodonato commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

fabiocos commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

silviodonato commented Nov 20, 2020

makortel commented Nov 23, 2020

makortel commented Nov 25, 2020

bsunanda commented Nov 25, 2020 via email

makortel commented Nov 25, 2020

silviodonato commented Dec 4, 2020

cvuosalo commented Dec 9, 2020

dan131riley commented Jan 19, 2021

Two relvals failing in Geometry #32181

Two relvals failing in Geometry #32181

Comments

mrodozov commented Nov 19, 2020

mrodozov commented Nov 19, 2020

cmsbuild commented Nov 19, 2020

cmsbuild commented Nov 19, 2020

makortel commented Nov 19, 2020

civanch commented Nov 19, 2020

silviodonato commented Nov 19, 2020

silviodonato commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

makortel commented Nov 19, 2020

silviodonato commented Nov 19, 2020

silviodonato commented Nov 19, 2020

silviodonato commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

fabiocos commented Nov 19, 2020

cvuosalo commented Nov 19, 2020

silviodonato commented Nov 20, 2020

makortel commented Nov 23, 2020

makortel commented Nov 25, 2020

bsunanda commented Nov 25, 2020 via email

makortel commented Nov 25, 2020

silviodonato commented Dec 4, 2020

cvuosalo commented Dec 9, 2020

dan131riley commented Jan 19, 2021