Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crashes in workflows 534 and 536 (Sherpa) #39914

Closed
makortel opened this issue Oct 28, 2022 · 6 comments
Closed

Crashes in workflows 534 and 536 (Sherpa) #39914

makortel opened this issue Oct 28, 2022 · 6 comments
Labels
generators-pending pending-signatures

Comments

@makortel
Copy link
Contributor

Step 1 of workflows 534.0 and 536.0 are crashing starting from CMSSW_12_6_X_2022-10-27-2300.

Log of the crash 534.0

[cmsbuild74:28179:0:28851] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x8)
==== backtrace (tid:  28851) ====
 0  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libucs.so.0(ucs_handle_error+0x2a4) [0x2b6370366d24]
 1  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libucs.so.0(+0x2ff27) [0x2b6370366f27]
 2  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libucs.so.0(+0x301f6) [0x2b63703671f6]
 3  /lib64/libpthread.so.0(+0x12ce0) [0x2b63316ebce0]
 4  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x2f3b3d3) [0x2b633e6453d3]
 5  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x2f3dac6) [0x2b633e647ac6]
 6  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x2f3fe78) [0x2b633e649e78]
 7  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x2eca676) [0x2b633e5d4676]
 8  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x2ef0493) [0x2b633e5fa493]
 9  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x1bf87ae) [0x2b633d3027ae]
10  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x30681b8) [0x2b633e7721b8]
11  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x3072e40) [0x2b633e77ce40]
12  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x16df013) [0x2b633cde9013]
13  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x719d92) [0x2b633be23d92]
14  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x70a85d) [0x2b633be1485d]
15  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(+0x70b947) [0x2b633be15947]
16  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(_ZN14TClingCallFunc15compile_wrapperERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES7_b+0x2a) [0x2b633bd38d4a]
17  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(_ZN14TClingCallFunc12make_wrapperEv+0xdf) [0x2b633bd3fbbf]
18  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libCling.so(_ZN14TClingCallFunc8IFacePtrEv+0x14a) [0x2b633bd3ffea]
19  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so(_ZN3edm16FunctionWithDictC2EP7TMethod+0xa7) [0x2b632ef3a417]
20  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreReflection.so(_ZNK3edm12TypeWithDict20functionMemberByNameERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_b+0x481) [0x2b632ef412c1]
21  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(_ZN4reco10findMethodERKN3edm12TypeWithDictERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorISt7variantIJahstijlmdfS9_EESaISE_EERSG_PKcRi+0x2fa) [0x2b6358c6e22a]
22  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(_ZNK4reco6parser12MethodSetter4pushERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKSt6vectorISt7variantIJahstijlmdfS7_EESaISC_EEPKcb+0xc5) [0x2b6358c3d9e5]
23  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(_ZNK4reco6parser12MethodSetterclEPKcS3_+0x371) [0x2b6358c3ea11]
24  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa7cb9) [0x2b6358c68cb9]
25  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xaacb5) [0x2b6358c6bcb5]
26  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa989c) [0x2b6358c6a89c]
27  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa3782) [0x2b6358c64782]
28  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xabae6) [0x2b6358c6cae6]
29  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xabb3f) [0x2b6358c6cb3f]
30  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa898c) [0x2b6358c6998c]
31  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xaa3dc) [0x2b6358c6b3dc]
32  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa9e8c) [0x2b6358c6ae8c]
33  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0x9e623) [0x2b6358c5f623]
34  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa8fad) [0x2b6358c69fad]
35  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa8a9a) [0x2b6358c69a9a]
36  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa82cc) [0x2b6358c692cc]
37  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa865c) [0x2b6358c6965c]
38  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(+0xa0ae3) [0x2b6358c61ae3]
39  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libCommonToolsUtils.so(_ZN4reco6parser9cutParserERKN3edm12TypeWithDictERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERSt10shared_ptrINS0_12SelectorBaseEEb+0xe4) [0x2b6358c5c794]
40  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/pluginPhysicsToolsHepMCCandAlgos_plugins.so(+0x85c47) [0x2b6372161c47]
41  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/pluginPhysicsToolsHepMCCandAlgos_plugins.so(_ZN17GenParticlePruner7produceERN3edm5EventERKNS0_10EventSetupE+0x1099) [0x2b6372160909]
42  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm6stream21EDProducerAdaptorBase7doEventERKNS_19EventTransitionInfoEPNS_16ActivityRegistryEPKNS_20ModuleCallingContextE+0x13d) [0x2b632f27f80d]
43  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm7WorkerTINS_6stream21EDProducerAdaptorBaseEE6implDoERKNS_19EventTransitionInfoEPKNS_20ModuleCallingContextE+0x6f) [0x2b632f263daf]
44  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm16convertException4wrapIZNS_6Worker9runModuleINS_16OccurrenceTraitsINS_14EventPrincipalELNS_16BranchActionTypeE1EEEEEbRKNT_18TransitionInfoTypeENS_8StreamIDERKNS_13ParentContextEPKNS8_7ContextEEUlvE_EEDTclfp_EES8_+0x25) [0x2b632f1b3935]
45  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm6Worker27runModuleAfterAsyncPrefetchINS_16OccurrenceTraitsINS_14EventPrincipalELNS_16BranchActionTypeE1EEEEENSt15__exception_ptr13exception_ptrES7_RKNT_18TransitionInfoTypeENS_8StreamIDERKNS_13ParentContextEPKNS8_7ContextE+0x293) [0x2b632f1b3cb3]
46  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(+0x2311b8) [0x2b632f1b61b8]
47  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so(+0x7819) [0x2b632ef54819]
48  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libtbb.so.12(+0x1f51c) [0x2b6330a3251c]
49  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libtbb.so.12(+0x21654) [0x2b6330a34654]
50  /lib64/libpthread.so.0(+0x81cf) [0x2b63316e11cf]
51  /lib64/libc.so.6(clone+0x43) [0x2b6331932dd3]
=================================

Exception_Handler::SignalHandler: Signal (11) caught. 
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  Event 2 ( 46s elapsed / 38m 9s left ) -> ETA: Fri Oct 28 09:32  
  XS = 0 pb +- ( 6462.77 pb = -2.14748e+07 % )  
  0x2b633bd38d4a  in 'TClingCallFunc::compile_wrapper(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' (??:?)
  0x2b633bd3fbbf  in 'TClingCallFunc::make_wrapper()' (??:?)
  0x2b633bd3ffea  in 'TClingCallFunc::IFacePtr()' (??:?)
  0x2b632ef3a417  in 'edm::FunctionWithDict::FunctionWithDict(TMethod*)' (??:?)
    0x2b632ef412c1  in 'edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const' (??:?)
  0x2b6358c6e22a  in 'reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&)' (findMethod.cc:?)
  0x2b6358c3d9e5  in 'reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const' (MethodSetter.cc:?)
  0x2b6358c3ea11  in 'reco::parser::MethodSetter::operator()(char const*, char const*) const' (??:?)
    0x2b6358c5c794  in 'reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool)' (??:?)
  0x2b6372160909  in 'GenParticlePruner::produce(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b632f27f80d  in 'edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b632f263daf  in 'edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
    0x2b632f1b3935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b632f1b3cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  0x2b6331932dd3  in 'clone' (:?)
}   Cannot continue.
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  0x2b633bd38d4a  in 'TClingCallFunc::compile_wrapper(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' (??:?)
    0x2b633bd3fbbf  in 'TClingCallFunc::make_wrapper()' (??:?)
  0x2b633bd3ffea  in 'TClingCallFunc::IFacePtr()' (??:?)
  0x2b632ef3a417  in 'edm::FunctionWithDict::FunctionWithDict(TMethod*)' (??:?)
  0x2b632ef412c1  in 'edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const' (??:?)
  0x2b6358c6e22a  in 'reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&)' (findMethod.cc:?)
  0x2b6358c3d9e5  in 'reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const' (MethodSetter.cc:?)
  0x2b6358c3ea11  in 'reco::parser::MethodSetter::operator()(char const*, char const*) const' (??:?)
  0x2b6358c5c794  in 'reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool)' (??:?)
    0x2b6372160909  in 'GenParticlePruner::produce(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b632f27f80d  in 'edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b632f263daf  in 'edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
    0x2b632f1b3935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b632f1b3cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  Event 3 ( 1m 5s elapsed / 35m 5s left ) -> ETA: Fri Oct 28 09:29  
  XS = -3601.89 pb +- ( 6238.93 pb = -173.21 % )  
  0x2b6331932dd3  in 'clone' (:?)
}
  Exception_Handler::Terminate(): Pre-crash status saved to '/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_12_6_X_2022-10-27-2300/pyRelval/534.0_sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+HARVESTGEN/Status__Fri_Oct_28_08-54-26_2022'.
Thanks for using LHAPDF 6.4.0. Please make sure to cite the paper:
  Eur.Phys.J. C75 (2015) 3, 132  (http://arxiv.org/abs/1412.7420)
  Event 3 ( 66 s total ) = 3920.74 evts/day                    
Event_Handler::MPISync() {
  Memory usage increased by 16 MB, now 1780 MB.
  This might indicate a memory leak!
  Please monitor this process closely.
}
In Event_Handler::Finish : Summarizing the run may take some time.
+---------------------------------------------------------+
|                                                         |
|  Total XS is -3601.89 pb +- ( 6238.93 pb = -173.21 % )  |
|                                                         |
+---------------------------------------------------------+
------------------------------------------------------------------------
Please cite the publications listed in 'Sherpa_References.tex'.
  Extract the bibtex list by running 'get_bibtex Sherpa_References.tex'
  or email the file to 'slaclib2@slac.stanford.edu', subject 'generate'.
------------------------------------------------------------------------
Exception_Handler::Exit: Exiting Sherpa with code (2)
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  0x2b6366d9fdd0  in 'ATOOLS::Terminate()' (Exception_Handler.C:140)
  0x2b63311d6977  in '__gxx_personality_v0' (eh_personality.cc:685)
  0x2b63316cfe8e  in '_Unwind_Resume' (unwind.inc:245)
  0x2b63300ff0f1  in 'TList::~TList()' (??:?)
  0x2b632fbc130c  in 'TFile::Close(char const*)' (??:?)
  0x2b6330049c8d  in 'TROOT::CloseFiles()' (??:?)
  0x2b633194a370  in 'on_exit' (:?)
  0x2b633bd38d4a  in 'TClingCallFunc::compile_wrapper(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' (??:?)
  0x2b633bd3fbbf  in 'TClingCallFunc::make_wrapper()' (??:?)
  0x2b633bd3ffea  in 'TClingCallFunc::IFacePtr()' (??:?)
  0x2b632ef3a417  in 'edm::FunctionWithDict::FunctionWithDict(TMethod*)' (??:?)
  0x2b632ef412c1  in 'edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const' (??:?)
  0x2b6358c6e22a  in 'reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&)' (findMethod.cc:?)
  0x2b6358c3d9e5  in 'reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const' (MethodSetter.cc:?)
  0x2b6358c3ea11  in 'reco::parser::MethodSetter::operator()(char const*, char const*) const' (??:?)
  0x2b6358c5c794  in 'reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool)' (??:?)
  0x2b6372160909  in 'GenParticlePruner::produce(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b632f27f80d  in 'edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b632f263daf  in 'edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
  0x2b632f1b3935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b632f1b3cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  0x2b6331932dd3  in 'clone' (:?)
}
Exception_Handler::Terminate(): Pre-crash status saved to '/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_12_6_X_2022-10-27-2300/pyRelval/534.0_sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+HARVESTGEN/Status__Fri_Oct_28_08-54-42_2022'.
------------------------------------------------------------------------
Please cite the publications listed in 'Sherpa_References.tex'.
  Extract the bibtex list by running 'get_bibtex Sherpa_References.tex'
  or email the file to 'slaclib2@slac.stanford.edu', subject 'generate'.
------------------------------------------------------------------------
Exception_Handler::Exit: Exiting Sherpa with code (2)
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  0x2b6366d9fdd0  in 'ATOOLS::Terminate()' (Exception_Handler.C:140)
  0x2b63311d6977  in '__gxx_personality_v0' (eh_personality.cc:685)
  0x2b63316cfe8e  in '_Unwind_Resume' (unwind.inc:245)
  0x2b63300ff0f1  in 'TList::~TList()' (??:?)
  0x2b632fbc130c  in 'TFile::Close(char const*)' (??:?)
  0x2b6330049c8d  in 'TROOT::CloseFiles()' (??:?)
  0x2b633194a370  in 'on_exit' (:?)
  0x2b6366d9fdd0  in 'ATOOLS::Terminate()' (Exception_Handler.C:140)
  0x2b63311d6977  in '__gxx_personality_v0' (eh_personality.cc:685)
  0x2b63316cfe8e  in '_Unwind_Resume' (unwind.inc:245)
  0x2b63300ff0f1  in 'TList::~TList()' (??:?)
  0x2b632fbc130c  in 'TFile::Close(char const*)' (??:?)
  0x2b6330049c8d  in 'TROOT::CloseFiles()' (??:?)
  0x2b633194a370  in 'on_exit' (:?)
  0x2b633bd38d4a  in 'TClingCallFunc::compile_wrapper(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)' (??:?)
  0x2b633bd3fbbf  in 'TClingCallFunc::make_wrapper()' (??:?)
  0x2b633bd3ffea  in 'TClingCallFunc::IFacePtr()' (??:?)
  0x2b632ef3a417  in 'edm::FunctionWithDict::FunctionWithDict(TMethod*)' (??:?)
  0x2b632ef412c1  in 'edm::TypeWithDict::functionMemberByName(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) const' (??:?)
  0x2b6358c6e22a  in 'reco::findMethod(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, char const*, int&)' (findMethod.cc:?)
  0x2b6358c3d9e5  in 'reco::parser::MethodSetter::push(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::variant<signed char, unsigned char, short, unsigned short, int, unsigned int, long, unsigned long, double, float, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, char const*, bool) const' (MethodSetter.cc:?)
  0x2b6358c3ea11  in 'reco::parser::MethodSetter::operator()(char const*, char const*) const' (??:?)
  0x2b6358c5c794  in 'reco::parser::cutParser(edm::TypeWithDict const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<reco::parser::SelectorBase>&, bool)' (??:?)
  0x2b6372160909  in 'GenParticlePruner::produce(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b632f27f80d  in 'edm::stream::EDProducerAdaptorBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b632f263daf  in 'edm::WorkerT<edm::stream::EDProducerAdaptorBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
  0x2b632f1b3935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b632f1b3cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  0x2b6331932dd3  in 'clone' (:?)
}
Exception_Handler::Terminate(): Pre-crash status saved to '/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_12_6_X_2022-10-27-2300/pyRelval/534.0_sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+HARVESTGEN/Status__Fri_Oct_28_08-54-56_2022'.

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_12_6_X_2022-10-27-2300/pyRelValMatrixLogs/run/534.0_sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+HARVESTGEN/step1_sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+sherpa_ZtoLL_2j_MEPSatNLO_13TeV_MASTER+HARVESTGEN.log#/

Log of the crash in 536.0

Begin processing the 5th record. Run 1, Event 5, LumiSection 1 on stream 3 at 28-Oct-2022 10:26:05.062 CEST
==== backtrace (tid:  16832) ====
 0  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libucs.so.0(ucs_handle_error+0x2a4) [0x2b9250ce6d24]
 1  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libucs.so.0(+0x2ff27) [0x2b9250ce6f27]
 2  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libucs.so.0(+0x301f6) [0x2b9250ce71f6]
 3  /lib64/libpthread.so.0(+0x12ce0) [0x2b9211c8cce0]
 4  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libComixPhasespace.so.0(_ZN5COMIX10PS_Channel15GenerateChannelEPN7METOOLS7CurrentERSt6vectorIPNS1_6VertexESaIS6_EE+0x32b) [0x2b924ed1716b]
 5  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libComixPhasespace.so.0(_ZN5COMIX10PS_Channel15GenerateChannelEPN7METOOLS7CurrentERSt6vectorIPNS1_6VertexESaIS6_EE+0x333) [0x2b924ed17173]
 6  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libComixPhasespace.so.0(_ZN5COMIX10PS_Channel15GenerateChannelERSt6vectorIPN7METOOLS6VertexESaIS4_EE+0x48) [0x2b924ed17248]
 7  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libComixPhasespace.so.0(_ZN5COMIX10PS_Channel13GeneratePointEPN6ATOOLS4Vec4IdEEPN6PHASIC8Cut_DataEPd+0x15f) [0x2b924ed1bedf]
 8  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libPhasicChannels.so.0(_ZN6PHASIC13Multi_Channel13GeneratePointEPN6ATOOLS4Vec4IdEEPNS_8Cut_DataE+0x18f) [0x2b924f2e08ff]
 9  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libPhasicMain.so.0(_ZN6PHASIC19Phase_Space_Handler12DifferentialEPNS_18Process_IntegratorENS_3psm4codeE+0x320) [0x2b924f35f5f0]
10  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libPhasicMain.so.0(_ZN6PHASIC19Phase_Space_Handler8OneEventEPNS_12Process_BaseEi+0x4b) [0x2b924f35fd6b]
11  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libPhasicProcess.so.0(_ZN6PHASIC14Single_Process8OneEventEii+0x4f) [0x2b924f3d083f]
12  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libPhasicProcess.so.0(_ZN6PHASIC15MCatNLO_Process8OneEventEii+0x3e7) [0x2b924f3e2877]
13  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libSherpaPerturbativePhysics.so.0(_ZN6SHERPA22Matrix_Element_Handler16GenerateOneEventEv+0x238) [0x2b924e82c7e8]
14  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libSherpaSingleEvents.so.0(_ZN6SHERPA16Signal_Processes5TreatEPN6ATOOLS9Blob_ListERd+0x1b6) [0x2b924e7d4e56]
15  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libSherpaSingleEvents.so.0(_ZN6SHERPA13Event_Handler18IterateEventPhasesERNS_9eventtype4codeERd+0x76) [0x2b924e7c8166]
16  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libSherpaSingleEvents.so.0(_ZN6SHERPA13Event_Handler33GenerateStandardPerturbativeEventERNS_9eventtype4codeE+0x52) [0x2b924e7cb512]
17  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libSherpaSingleEvents.so.0(_ZN6SHERPA13Event_Handler13GenerateEventENS_9eventtype4codeE+0x2f2) [0x2b924e7ccb12]
18  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libSherpaMain.so.0(_ZN6SHERPA6Sherpa16GenerateOneEventEb+0x99) [0x2b9247647299]
19  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/pluginGeneratorInterfaceSherpaInterface.so(_ZN16SherpaHadronizer27generatePartonsAndHadronizeEv+0x55) [0x2b9244b06c45]
20  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/pluginGeneratorInterfaceSherpaInterface.so(_ZN3edm15GeneratorFilterI16SherpaHadronizerN3gen19ExternalDecayDriverEE6filterERNS_5EventERKNS_10EventSetupE+0x1b2) [0x2b9244b121d2]
21  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm3one12EDFilterBase7doEventERKNS_19EventTransitionInfoEPNS_16ActivityRegistryEPKNS_20ModuleCallingContextE+0x110) [0x2b920f817dd0]
22  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm7WorkerTINS_3one12EDFilterBaseEE6implDoERKNS_19EventTransitionInfoEPKNS_20ModuleCallingContextE+0x6f) [0x2b920f8044ef]
23  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm16convertException4wrapIZNS_6Worker9runModuleINS_16OccurrenceTraitsINS_14EventPrincipalELNS_16BranchActionTypeE1EEEEEbRKNT_18TransitionInfoTypeENS_8StreamIDERKNS_13ParentContextEPKNS8_7ContextEEUlvE_EEDTclfp_EES8_+0x25) [0x2b920f754935]
24  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm6Worker27runModuleAfterAsyncPrefetchINS_16OccurrenceTraitsINS_14EventPrincipalELNS_16BranchActionTypeE1EEEEENSt15__exception_ptr13exception_ptrES7_RKNT_18TransitionInfoTypeENS_8StreamIDERKNS_13ParentContextEPKNS8_7ContextE+0x293) [0x2b920f754cb3]
25  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(+0x230729) [0x2b920f756729]
26  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm20SerialTaskQueueChain11actionToRunIRZNS_6Worker13RunModuleTaskINS_16OccurrenceTraitsINS_14EventPrincipalELNS_16BranchActionTypeE1EEEE7executeEvEUlvE_EEvOT_+0xf) [0x2b920f75774f]
27  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm15SerialTaskQueue10QueuedTaskIZNS_20SerialTaskQueueChain4pushIRZNS_6Worker13RunModuleTaskINS_16OccurrenceTraitsINS_14EventPrincipalELNS_16BranchActionTypeE1EEEE7executeEvEUlvE_EEvRN3tbb6detail2d110task_groupEOT_EUlvE_E7executeEv+0x11) [0x2b920f7577c1]
28  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreConcurrency.so(+0x6125) [0x2b920f4f4125]
29  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libtbb.so.12(+0x26ae9) [0x2b9210fdaae9]
30  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(+0x1af787) [0x2b920f6d5787]
31  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm14EventProcessor11processRunsEv+0x195) [0x2b920f6bd485]
32  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/lib/el8_amd64_gcc11/libFWCoreFramework.so(_ZN3edm14EventProcessor15runToCompletionEv+0x206) [0x2b920f6ca046]
33  cmsRun() [0x40a18d]
34  /cvmfs/cms-ib.cern.ch/nweek-02756/el8_amd64_gcc11/cms/cmssw/CMSSW_12_6_X_2022-10-27-2300/external/el8_amd64_gcc11/lib/libtbb.so.12(+0x148a8) [0x2b9210fc88a8]
35  cmsRun() [0x40afd9]
36  cmsRun(main+0x14c) [0x4096ec]
37  /lib64/libc.so.6(__libc_start_main+0xf3) [0x2b9211ed4cf3]
38  cmsRun(_start+0x2e) [0x4097fe]
=================================

Exception_Handler::SignalHandler: Signal (11) caught. 
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  0x2b924ed1716b  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17173  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17248  in 'COMIX::PS_Channel::GenerateChannel(std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:605)
  0x2b924ed1bedf  in 'COMIX::PS_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*, double*)' (PS_Channel.C:646)
  0x2b924f2e08ff  in 'PHASIC::Multi_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*)' (Multi_Channel.C:319)
  0x2b924f35f5f0  in 'PHASIC::Phase_Space_Handler::Differential(PHASIC::Process_Integrator*, PHASIC::psm::code)' (Process_Integrator.H:129)
  0x2b924f35fd6b  in 'PHASIC::Phase_Space_Handler::OneEvent(PHASIC::Process_Base*, int)' (Phase_Space_Handler.C:407)
  0x2b924f3d083f  in 'PHASIC::Single_Process::OneEvent(int, int)' (Smart_Pointer.H:72)
  0x2b924f3e2877  in 'PHASIC::MCatNLO_Process::OneEvent(int, int)' (MCatNLO_Process.C:577)
  0x2b924e82c7e8  in 'SHERPA::Matrix_Element_Handler::GenerateOneEvent()' (Matrix_Element_Handler.C:211)
  0x2b924e7d4e56  in 'SHERPA::Signal_Processes::Treat(ATOOLS::Blob_List*, double&)' (Signal_Processes.C:66)
  0x2b924e7c8166  in 'SHERPA::Event_Handler::IterateEventPhases(SHERPA::eventtype::code&, double&)' (Event_Handler.C:205)
  0x2b924e7cb512  in 'SHERPA::Event_Handler::GenerateStandardPerturbativeEvent(SHERPA::eventtype::code&)' (Event_Handler.C:307 (discriminator 1))
  0x2b924e7ccb12  in 'SHERPA::Event_Handler::GenerateEvent(SHERPA::eventtype::code)' (Event_Handler.C:127)
  0x2b9247647299  in 'SHERPA::Sherpa::GenerateOneEvent(bool)' (Sherpa.C:232)
  0x2b9244b06c45  in 'SherpaHadronizer::generatePartonsAndHadronize()' (??:?)
  0x2b9244b121d2  in 'edm::GeneratorFilter<SherpaHadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b920f817dd0  in 'edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f8044ef  in 'edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f754935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b920f754cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  0x2b920f75774f  in 'void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)' (??:?)
  0x2b920f7577c1  in 'edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute()' (??:?)
  0x2b920f6bd485  in 'edm::EventProcessor::processRuns()' (??:?)
  0x2b920f6ca046  in 'edm::EventProcessor::runToCompletion()' (??:?)
  0x4096ec        in 'main' 
}   Cannot continue.
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  0x2b924ed1716b  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17173  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17248  in 'COMIX::PS_Channel::GenerateChannel(std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:605)
  0x2b924ed1bedf  in 'COMIX::PS_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*, double*)' (PS_Channel.C:646)
  0x2b924f2e08ff  in 'PHASIC::Multi_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*)' (Multi_Channel.C:319)
  0x2b924f35f5f0  in 'PHASIC::Phase_Space_Handler::Differential(PHASIC::Process_Integrator*, PHASIC::psm::code)' (Process_Integrator.H:129)
  0x2b924f35fd6b  in 'PHASIC::Phase_Space_Handler::OneEvent(PHASIC::Process_Base*, int)' (Phase_Space_Handler.C:407)
  0x2b924f3d083f  in 'PHASIC::Single_Process::OneEvent(int, int)' (Smart_Pointer.H:72)
  0x2b924f3e2877  in 'PHASIC::MCatNLO_Process::OneEvent(int, int)' (MCatNLO_Process.C:577)
  0x2b924e82c7e8  in 'SHERPA::Matrix_Element_Handler::GenerateOneEvent()' (Matrix_Element_Handler.C:211)
  0x2b924e7d4e56  in 'SHERPA::Signal_Processes::Treat(ATOOLS::Blob_List*, double&)' (Signal_Processes.C:66)
  0x2b924e7c8166  in 'SHERPA::Event_Handler::IterateEventPhases(SHERPA::eventtype::code&, double&)' (Event_Handler.C:205)
  0x2b924e7cb512  in 'SHERPA::Event_Handler::GenerateStandardPerturbativeEvent(SHERPA::eventtype::code&)' (Event_Handler.C:307 (discriminator 1))
  0x2b924e7ccb12  in 'SHERPA::Event_Handler::GenerateEvent(SHERPA::eventtype::code)' (Event_Handler.C:127)
  0x2b9247647299  in 'SHERPA::Sherpa::GenerateOneEvent(bool)' (Sherpa.C:232)
  0x2b9244b06c45  in 'SherpaHadronizer::generatePartonsAndHadronize()' (??:?)
  0x2b9244b121d2  in 'edm::GeneratorFilter<SherpaHadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b920f817dd0  in 'edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f8044ef  in 'edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f754935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b920f754cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  0x2b920f75774f  in 'void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)' (??:?)
  0x2b920f7577c1  in 'edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute()' (??:?)
  0x2b920f6bd485  in 'edm::EventProcessor::processRuns()' (??:?)
  0x2b920f6ca046  in 'edm::EventProcessor::runToCompletion()' (??:?)
  0x4096ec        in 'main' 
}
Exception_Handler::Terminate(): Pre-crash status saved to '/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_12_6_X_2022-10-27-2300/pyRelval/536.0_sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+HARVESTGEN/Status__Fri_Oct_28_10-26-13_2022'.
  Event 0 ( 9 s total ) = 0 evts/day                    
In Event_Handler::Finish : Summarizing the run may take some time.
+-------------------------------------------------+
|                                                 |
|  Total XS is 0 pb +- ( 0 pb = -2.14748e+07 % )  |
|                                                 |
+-------------------------------------------------+
------------------------------------------------------------------------
Please cite the publications listed in 'Sherpa_References.tex'.
  Extract the bibtex list by running 'get_bibtex Sherpa_References.tex'
  or email the file to 'slaclib2@slac.stanford.edu', subject 'generate'.
------------------------------------------------------------------------
Exception_Handler::Exit: Exiting Sherpa with code (2)
Thanks for using LHAPDF 6.4.0. Please make sure to cite the paper:
  Eur.Phys.J. C75 (2015) 3, 132  (http://arxiv.org/abs/1412.7420)
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  0x2b924774edd0  in 'ATOOLS::Terminate()' (Exception_Handler.C:140)
  0x2b9211777977  in '__gxx_personality_v0' (eh_personality.cc:685)
  0x2b9211c70e8e  in '_Unwind_Resume' (unwind.inc:245)
  0x2b92106a00f1  in 'TList::~TList()' (??:?)
  0x2b921016230c  in 'TFile::Close(char const*)' (??:?)
  0x2b92105eac8d  in 'TROOT::CloseFiles()' (??:?)
  0x2b9211eeb370  in 'on_exit' (:?)
  0x2b924ed1716b  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17173  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17248  in 'COMIX::PS_Channel::GenerateChannel(std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:605)
  0x2b924ed1bedf  in 'COMIX::PS_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*, double*)' (PS_Channel.C:646)
  0x2b924f2e08ff  in 'PHASIC::Multi_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*)' (Multi_Channel.C:319)
  0x2b924f35f5f0  in 'PHASIC::Phase_Space_Handler::Differential(PHASIC::Process_Integrator*, PHASIC::psm::code)' (Process_Integrator.H:129)
  0x2b924f35fd6b  in 'PHASIC::Phase_Space_Handler::OneEvent(PHASIC::Process_Base*, int)' (Phase_Space_Handler.C:407)
  0x2b924f3d083f  in 'PHASIC::Single_Process::OneEvent(int, int)' (Smart_Pointer.H:72)
  0x2b924f3e2877  in 'PHASIC::MCatNLO_Process::OneEvent(int, int)' (MCatNLO_Process.C:577)
  0x2b924e82c7e8  in 'SHERPA::Matrix_Element_Handler::GenerateOneEvent()' (Matrix_Element_Handler.C:211)
  0x2b924e7d4e56  in 'SHERPA::Signal_Processes::Treat(ATOOLS::Blob_List*, double&)' (Signal_Processes.C:66)
  0x2b924e7c8166  in 'SHERPA::Event_Handler::IterateEventPhases(SHERPA::eventtype::code&, double&)' (Event_Handler.C:205)
  0x2b924e7cb512  in 'SHERPA::Event_Handler::GenerateStandardPerturbativeEvent(SHERPA::eventtype::code&)' (Event_Handler.C:307 (discriminator 1))
  0x2b924e7ccb12  in 'SHERPA::Event_Handler::GenerateEvent(SHERPA::eventtype::code)' (Event_Handler.C:127)
  0x2b9247647299  in 'SHERPA::Sherpa::GenerateOneEvent(bool)' (Sherpa.C:232)
  0x2b9244b06c45  in 'SherpaHadronizer::generatePartonsAndHadronize()' (??:?)
  0x2b9244b121d2  in 'edm::GeneratorFilter<SherpaHadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b920f817dd0  in 'edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f8044ef  in 'edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f754935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b920f754cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  0x2b920f75774f  in 'void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)' (??:?)
  0x2b920f7577c1  in 'edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute()' (??:?)
  0x2b920f6bd485  in 'edm::EventProcessor::processRuns()' (??:?)
  0x2b920f6ca046  in 'edm::EventProcessor::runToCompletion()' (??:?)
  0x4096ec        in 'main' 
}
Exception_Handler::Terminate(): Pre-crash status saved to '/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_12_6_X_2022-10-27-2300/pyRelval/536.0_sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+HARVESTGEN/Status__Fri_Oct_28_10-26-18_2022'.
------------------------------------------------------------------------
Please cite the publications listed in 'Sherpa_References.tex'.
  Extract the bibtex list by running 'get_bibtex Sherpa_References.tex'
  or email the file to 'slaclib2@slac.stanford.edu', subject 'generate'.
------------------------------------------------------------------------
Exception_Handler::Exit: Exiting Sherpa with code (2)
Exception_Handler::GenerateStackTrace(..): Generating stack trace 
{
  0x2b924774edd0  in 'ATOOLS::Terminate()' (Exception_Handler.C:140)
  0x2b9211777977  in '__gxx_personality_v0' (eh_personality.cc:685)
  0x2b9211c70e8e  in '_Unwind_Resume' (unwind.inc:245)
  0x2b92106a00f1  in 'TList::~TList()' (??:?)
  0x2b921016230c  in 'TFile::Close(char const*)' (??:?)
  0x2b92105eac8d  in 'TROOT::CloseFiles()' (??:?)
  0x2b9211eeb370  in 'on_exit' (:?)
  0x2b924774edd0  in 'ATOOLS::Terminate()' (Exception_Handler.C:140)
  0x2b9211777977  in '__gxx_personality_v0' (eh_personality.cc:685)
  0x2b9211c70e8e  in '_Unwind_Resume' (unwind.inc:245)
  0x2b92106a00f1  in 'TList::~TList()' (??:?)
  0x2b921016230c  in 'TFile::Close(char const*)' (??:?)
  0x2b92105eac8d  in 'TROOT::CloseFiles()' (??:?)
  0x2b9211eeb370  in 'on_exit' (:?)
  0x2b924ed1716b  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17173  in 'COMIX::PS_Channel::GenerateChannel(METOOLS::Current*, std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:594 (discriminator 1))
  0x2b924ed17248  in 'COMIX::PS_Channel::GenerateChannel(std::vector<METOOLS::Vertex*, std::allocator<METOOLS::Vertex*> >&)' (PS_Channel.C:605)
  0x2b924ed1bedf  in 'COMIX::PS_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*, double*)' (PS_Channel.C:646)
  0x2b924f2e08ff  in 'PHASIC::Multi_Channel::GeneratePoint(ATOOLS::Vec4<double>*, PHASIC::Cut_Data*)' (Multi_Channel.C:319)
  0x2b924f35f5f0  in 'PHASIC::Phase_Space_Handler::Differential(PHASIC::Process_Integrator*, PHASIC::psm::code)' (Process_Integrator.H:129)
  0x2b924f35fd6b  in 'PHASIC::Phase_Space_Handler::OneEvent(PHASIC::Process_Base*, int)' (Phase_Space_Handler.C:407)
  0x2b924f3d083f  in 'PHASIC::Single_Process::OneEvent(int, int)' (Smart_Pointer.H:72)
  0x2b924f3e2877  in 'PHASIC::MCatNLO_Process::OneEvent(int, int)' (MCatNLO_Process.C:577)
  0x2b924e82c7e8  in 'SHERPA::Matrix_Element_Handler::GenerateOneEvent()' (Matrix_Element_Handler.C:211)
  0x2b924e7d4e56  in 'SHERPA::Signal_Processes::Treat(ATOOLS::Blob_List*, double&)' (Signal_Processes.C:66)
  0x2b924e7c8166  in 'SHERPA::Event_Handler::IterateEventPhases(SHERPA::eventtype::code&, double&)' (Event_Handler.C:205)
  0x2b924e7cb512  in 'SHERPA::Event_Handler::GenerateStandardPerturbativeEvent(SHERPA::eventtype::code&)' (Event_Handler.C:307 (discriminator 1))
  0x2b924e7ccb12  in 'SHERPA::Event_Handler::GenerateEvent(SHERPA::eventtype::code)' (Event_Handler.C:127)
  0x2b9247647299  in 'SHERPA::Sherpa::GenerateOneEvent(bool)' (Sherpa.C:232)
  0x2b9244b06c45  in 'SherpaHadronizer::generatePartonsAndHadronize()' (??:?)
  0x2b9244b121d2  in 'edm::GeneratorFilter<SherpaHadronizer, gen::ExternalDecayDriver>::filter(edm::Event&, edm::EventSetup const&)' (??:?)
  0x2b920f817dd0  in 'edm::one::EDFilterBase::doEvent(edm::EventTransitionInfo const&, edm::ActivityRegistry*, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f8044ef  in 'edm::WorkerT<edm::one::EDFilterBase>::implDo(edm::EventTransitionInfo const&, edm::ModuleCallingContext const*)' (??:?)
  0x2b920f754935  in 'decltype ({parm#1}()) edm::convertException::wrap<edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1}>(edm::Worker::runModule<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)::{lambda()#1})' (??:?)
  0x2b920f754cb3  in 'std::__exception_ptr::exception_ptr edm::Worker::runModuleAfterAsyncPrefetch<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >(std::__exception_ptr::exception_ptr, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::TransitionInfoType const&, edm::StreamID, edm::ParentContext const&, edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1>::Context const*)' (??:?)
  0x2b920f75774f  in 'void edm::SerialTaskQueueChain::actionToRun<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)' (??:?)
  0x2b920f7577c1  in 'edm::SerialTaskQueue::QueuedTask<edm::SerialTaskQueueChain::push<edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&>(tbb::detail::d1::task_group&, edm::Worker::RunModuleTask<edm::OccurrenceTraits<edm::EventPrincipal, (edm::BranchActionType)1> >::execute()::{lambda()#1}&)::{lambda()#1}>::execute()' (??:?)
  0x2b920f6bd485  in 'edm::EventProcessor::processRuns()' (??:?)
  0x2b920f6ca046  in 'edm::EventProcessor::runToCompletion()' (??:?)
  0x4096ec        in 'main' 
}
Exception_Handler::Terminate(): Pre-crash status saved to '/data/cmsbld/jenkins/workspace/ib-run-relvals/CMSSW_12_6_X_2022-10-27-2300/pyRelval/536.0_sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+HARVESTGEN/Status__Fri_Oct_28_10-26-25_2022'.

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc11/CMSSW_12_6_X_2022-10-27-2300/pyRelValMatrixLogs/run/536.0_sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+HARVESTGEN/step1_sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+sherpa_ttbar_2j_MENLOPS_13TeV_MASTER+HARVESTGEN.log#/
@makortel
Copy link
Contributor Author

assign generators

@cmsbuild
Copy link
Contributor

New categories assigned: generators

@mkirsano,@menglu21,@alberto-sanchez,@SiewYan,@GurpreetSinghChahal,@Saptaparna you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @rappoccio, @makortel, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

@makortel
Copy link
Contributor Author

Something to note in both cases the crash report comes presumably from Sherpa rather than CMSSW (IB dashboard also reports these jobs to exit with SIGKILL).

I don't see anything immediately obvious merged in CMSSW_12_6_X_2022-10-27-2300 that would cause the failures. The updates of Rivet and YODA (cms-sw/cmsdist#8109) are related to generators, but can they play any role here?

@smuzaffar smuzaffar mentioned this issue Nov 1, 2022
Merged
@smuzaffar
Copy link
Contributor

yes this couldbe related to rivet/yoda update. I am testing a revert of cms-sw/cmsdist#8109 it here

@smuzaffar
Copy link
Contributor

cms-sw/cmsdist#8163 (reverts rivet/yoda update) has been merged and looks like this has fixed the workflow 534/536 issue. @mseidel42 is looking in to if Sherpa-internal Rivet interface n CMSSW is really needed

@mseidel42 mseidel42 mentioned this issue Nov 24, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
generators-pending pending-signatures
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@makortel @smuzaffar @cmsbuild