Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DQM/Integration unit test failure #31023

Closed
makortel opened this issue Aug 3, 2020 · 16 comments
Closed

DQM/Integration unit test failure #31023

makortel opened this issue Aug 3, 2020 · 16 comments

Comments

@makortel
Copy link
Contributor

makortel commented Aug 3, 2020

A unit test in DQM/Integration has been failing since CMSSW_11_2_X_2020-07-28-2300 with

...
In module 'std' imported from input_line_1:1:
/cvmfs/cms-ib.cern.ch/nweek-02640/slc7_amd64_gcc820/external/gcc/8.2.0-bcolbf/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.4.0/../../../../include/c++/8.4.0/bits/stl_vector.h:312:16: error: no member named '_M_finish' in 'std::_Vector_base<gem::GEBdata, std::allocator<gem::GEBdata> >::_Vector_impl'
        this->_M_impl._M_finish = this->_M_impl._M_start;
        ~~~~~~~~~~~~~ ^
/cvmfs/cms-ib.cern.ch/nweek-02640/slc7_amd64_gcc820/external/gcc/8.2.0-bcolbf/bin/../lib/gcc/x86_64-unknown-linux-gnu/8.4.0/../../../../include/c++/8.4.0/bits/stl_vector.h:313:16: error: no member named '_M_end_of_storage' in 'std::_Vector_base<gem::GEBdata, std::allocator<gem::GEBdata> >::_Vector_impl'
        this->_M_impl._M_end_of_storage = this->_M_impl._M_start + __n;
        ~~~~~~~~~~~~~ ^
03-Aug-2020 00:08:29 CEST  Closed file root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/express/Commissioning2019/ExpressCosmics/FEVT/Express-v1/000/334/393/00000/D0F052ED-9CA5-F547-BA73-2AA370D51AE8.root
----- Begin Fatal Exception 03-Aug-2020 00:08:30 CEST-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GEMRawToDigiModule label='muonGEMDigis'
   Additional Info:
      [a] Fatal Root Error: @SUB=TInterpreter::AutoParse
Error parsing payload code for class gem::AMCdata with content:

#line 1 "DataFormatsGEMDigi_xr dictionary payload"
...

(see e.g. https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc820/CMSSW_11_2_X_2020-08-02-2300/unitTestLogs/DQM/Integration#/15373-15373 for full, long, log)

@makortel
Copy link
Contributor Author

makortel commented Aug 3, 2020

assign core, dqm

@cmsbuild
Copy link
Contributor

cmsbuild commented Aug 3, 2020

New categories assigned: core,dqm

@Dr15Jones,@smuzaffar,@jfernan2,@andrius-k,@fioriNTU,@makortel,@kmaeshima you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

cmsbuild commented Aug 3, 2020

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

makortel commented Aug 3, 2020

I did not spot anything obvious from the merged PRs between CMSSW_11_2_X_2020-07-28-1100 and CMSSW_11_2_X_2020-07-28-2300.

I did notice that CMSSW_11_2_X_2020-07-28-2300 had an update on ROOT 6.20 cms-sw/cmsdist#6112. I wonder if that could be related.

@pcanal Could this error be related to an update in ROOT?

@jfernan2
Copy link
Contributor

jfernan2 commented Aug 3, 2020

@hyunyong @seungjin-yang GEM DQM contacts, can you please have a look too? Thanks

@makortel
Copy link
Contributor Author

makortel commented Aug 3, 2020

Here is a stack trace of the exception

#0  0x00007ffff59a2d2d in __cxxabiv1::__cxa_throw (obj=0x7fffc4a04180, tinfo=0x7ffff77cbc58 <typeinfo for edm::Exception>, dest=0x7ffff776ef70 <edm::Exception::~Exception()>) at ../../../../libstdc++-v3/libsupc++/eh_throw.cc:78
#1  0x00007fffef5d9303 in (anonymous namespace)::RootErrorHandlerImpl(int, char const*, char const*) [clone .cold.374] () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreServicesPlugins.so
#2  0x00007ffff6ad279e in ErrorHandler () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#3  0x00007ffff6ae61c4 in TObject::Error(char const*, char const*, ...) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#4  0x00007fffeb4ee13e in TCling::AutoParseImplRecurse(char const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCling.so
#5  0x00007fffeb4eef33 in TCling::AutoParse(char const*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCling.so
#6  0x00007ffff6b8bc6a in TClass::LoadClassInfo() const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#7  0x00007ffff6b8c71d in TClass::CallShowMembers(void const*, TMemberInspector&, bool) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#8  0x00007ffff6ae2cf6 in TMemberInspector::InspectMember(char const*, void const*, char const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#9  0x00007fffeb4f6c5a in TCling::InspectMembers(TMemberInspector&, void const*, TClass const*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCling.so
#10 0x00007ffff6b8c70d in TClass::CallShowMembers(void const*, TMemberInspector&, bool) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#11 0x00007ffff6b9a4e5 in TClass::BuildRealData(void*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#12 0x00007ffff7055682 in TStreamerInfo::Build() () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libRIO.so
#13 0x00007ffff6b8b5d6 in TClass::GetStreamerInfo(int) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#14 0x00007ffff6b97220 in TClass::GetMissingDictionariesForPairElements(TCollection&, TCollection&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#15 0x00007ffff6b96e7a in TClass::GetMissingDictionariesWithRecursionCheck(TCollection&, TCollection&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#16 0x00007ffff6b96dd1 in TClass::GetMissingDictionariesWithRecursionCheck(TCollection&, TCollection&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#17 0x00007ffff6b97112 in TClass::GetMissingDictionariesForMembers(TCollection&, TCollection&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#18 0x00007ffff6b96f80 in TClass::GetMissingDictionariesWithRecursionCheck(TCollection&, TCollection&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#19 0x00007ffff6b97112 in TClass::GetMissingDictionariesForMembers(TCollection&, TCollection&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#20 0x00007ffff6b9ec2d in TClass::GetMissingDictionaries(THashTable&, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/external/slc7_amd64_gcc820/lib/libCore.so
#21 0x00007ffff7e4c8a4 in edm::checkClassDictionaries(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, edm::TypeWithDict const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/lib/slc7_amd64_gcc820/libFWCoreReflection.so
#22 0x00007ffff7c6d6e5 in edm::ProductRegistryHelper::addToRegistry(__gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, __gnu_cxx::__normal_iterator<edm::ProductRegistryHelper::TypeLabelItem const*, std::vector<edm::ProductRegistryHelper::TypeLabelItem, std::allocator<edm::ProductRegistryHelper::TypeLabelItem> > > const&, edm::ModuleDescription const&, edm::ProductRegistry&, edm::ProductRegistryHelper*, bool) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#23 0x00007ffff7c6b3cc in edm::ProducerBase::registerProducts(edm::ProducerBase*, edm::ProductRegistry*, edm::ModuleDescription const&) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#24 0x00007fffd059e0e7 in edm::maker::ModuleHolderT<edm::global::EDProducerBase>::registerProductsAndCallbacks(edm::ProductRegistry*) () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/lib/slc7_amd64_gcc820/pluginFWCoreModules.so
#25 0x00007ffff7ce2253 in edm::Maker::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so
#26 0x00007ffff7c2eb15 in edm::Factory::makeModule(edm::MakeModuleParams const&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&, edm::signalslot::Signal<void (edm::ModuleDescription const&)>&) const () from /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw/CMSSW_11_2_X_2020-08-03-1100/lib/slc7_amd64_gcc820/libFWCoreFramework.so

@Dr15Jones
Copy link
Contributor

Using the following trivial configuration

import FWCore.ParameterSet.Config as cms
process = cms.Process("TEST")
process.source = cms.Source("EmptySource")
process.maxEvents.input = 10
process.t = cms.EDProducer("GEMRawToDigiModule")
process.p = cms.Path(process.t)

The problem does not happen and the module loads fine. So the problem must be an iteraction with more than just the dictionary load.

@Dr15Jones
Copy link
Contributor

Using the follow not so trivial configuration

import FWCore.ParameterSet.Config as cms
process = cms.Process("TEST")
process.source = cms.Source("PoolSource", fileNames = cms.untracked.vstring("/store/user/cmsbuild/store/express/Commissioning2019/ExpressCosmics/FEVT/Express-v1/000/334/393/00000/D0F052ED-9CA5-F547-BA73-2AA370D51AE8.root"))
process.maxEvents.input = 10
process.t = cms.EDProducer("GEMRawToDigiModule")
process.p = cms.Path(process.t)

Also succeeded which means it is not a simple bad interaction with the file and the dictionaries.

@makortel
Copy link
Contributor Author

makortel commented Aug 4, 2020

Using the following trivial configuration
...
The problem does not happen and the module loads fine. So the problem must be an iteraction with more than just the dictionary load.

That was only because the offending produces() call is not called by default. Enabling that reproduces the problem

import FWCore.ParameterSet.Config as cms
process = cms.Process("TEST")
process.source = cms.Source("EmptySource")
process.maxEvents.input = 10
process.t = cms.EDProducer("GEMRawToDigiModule",
    unPackStatusDigis = cms.bool(True)
)
process.p = cms.Path(process.t)

@Dr15Jones
Copy link
Contributor

So if I add

process.add_(cms.Service("InitRootHandlers", DebugLevel = cms.untracked.int32(10)))

to the minimum failure configuration I get a different exception

----- Begin Fatal Exception 04-Aug-2020 10:27:11 CDT-----------------------
An exception of category 'FatalRootError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=GEMRawToDigiModule label='muonGEMDigis'
   Additional Info:
      [a] Fatal Root Error: @SUB=GetClassSharedLibsForModule
Decl found for pair<unsigned short,vector<gem::AMCdata> > is not part of a module

----- End Fatal Exception -------------------------------------------------

@watson-ij
Copy link
Contributor

watson-ij commented Aug 4, 2020

I was modifying the GEM classes recently so I took a look. is in the classes_def.xml in DataFormats/GEMDigi, but this pair<> isnt. Looking around, this is what the other classes_def.xml also do, so I'm not sure why it would be an issue only for us in GEM? Could it be an the issue with the gem:: namespace? Or am I reading the error message wrong and its another issue with the class?

@Dr15Jones
Copy link
Contributor

Fixed in #31048

@makortel
Copy link
Contributor Author

makortel commented Aug 4, 2020

Thanks @Dr15Jones. I'm left to wonder to things (mostly for @pcanal I presume)

  • Why we did not get a complaint from missing dictionaries before? (perhaps they surfaced now thanks to an update in ROOT?)
  • Why did it cause an error inside ROOT instead of the usual report of a missing dictionary?

@Dr15Jones
Copy link
Contributor

I see that the other uses of MuonDigiCollection<> do explicitly generate dictionaries for the relevant std::pair

<class name="std::pair<CSCDetId,std::vector<CSCWireDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCRPCDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCStripDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCComparatorDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCCorrelatedLCTDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<GEMCSCLCTDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCCLCTDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCCLCTPreTriggerDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCALCTDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCALCTPreTriggerDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCCFEBStatusDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCTMBStatusDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCDCCFormatStatusDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCDMBStatusDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCDDUStatusDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCDCCStatusDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCALCTStatusDigi> >"/>
<class name="std::pair<CSCDetId,std::vector<CSCCLCTPreTrigger> >"/>

@silviodonato
Copy link
Contributor

solved in CMSSW_11_2_X_2020-08-05-2300 (#31048)

@makortel
Copy link
Contributor Author

makortel commented Aug 6, 2020

+1

(although I'm still curious of #31023 (comment))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants