Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug fixes for EventSetup consumes corner cases #26550

Merged
merged 1 commit into from Apr 27, 2019

Conversation

wddgit
Copy link
Contributor

@wddgit wddgit commented Apr 26, 2019

PR description:

Fixes 2 bugs. One causes an assert failure and the other an out
of bounds read. These bugs were only introduced recently and
only in 10_6_X. The assert occurs for ESProducers that after a
call to setWhatProduced for multiple products call setWhatProduced
another time. The out of bounds read failure occurs when
esConsumes is called for multiple types of transitions (event,
beginRun, ...) in the same producer, filter, or analyzer.

I am not aware of any existing code in CMSSW that actually
hits these corner cases. As far as I know this will not affect the output
of any existing tests.

PR validation:

Modifies existing unit tests so the assert bug would now fail
a unit test. Modified another test to hit the out of bounds read,
although you need valgrind to actually see it with this test, but
there is a new and more complex test coming soon in the PR
that implements concurrent IOVs that fails when the out of bounds
read bug is hit. These bugs were noticed while testing the concurrent
IOV PR.

Fixes 2 bugs. One causes an assert failure and the other an out
of bounds read. These bugs were only introduced recently and
only in 10_6_X. I am not aware of any existing code that actually
hits these corner cases. The assert occurs after a call to
setWhatProduced for multiple products followed by another call
to setWhatProduced. The out of bounds read failure occurs when
esConsumes is called for mutiple types of transitions (event,
beginRun, ...) in the same producer, filter, or analyzer.

Modifies existing unit tests so the assert bug would now fail
a unit test. Modified another test to hit the out of bounds read,
although you need valgrind to actually see it with this test, but
there is a more involved tests coming soon in the PR that implements
concurrent IOVs that fails with the out of bounds read bug. These
bugs were noticed while testing the concurrent IOV PR.
@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-26550/9449

  • This PR adds an extra 40KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @wddgit (W. David Dagenhart) for master.

It involves the following packages:

FWCore/Framework

@cmsbuild, @smuzaffar, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

@wddgit
Copy link
Contributor Author

wddgit commented Apr 26, 2019

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Apr 26, 2019

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34370/console Started: 2019/04/26 20:42
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/34374/console Started: 2019/04/26 21:50

@Dr15Jones
Copy link
Contributor

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

@cmsbuild
Copy link
Contributor

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-26550/34374/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 3 differences found in the comparisons
  • DQMHistoTests: Total files compared: 33
  • DQMHistoTests: Total histograms compared: 3211964
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3211759
  • DQMHistoTests: Total skipped: 204
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
  • Checked 137 log files, 14 edm output root files, 33 DQM output files

@fabiocos
Copy link
Contributor

@Dr15Jones in the header check there is a list of reported errors that seem unrelated to this specific PR, but seem also genuine (i.e. not the usual internal compiler failure). Just to draw your attention to that:

>> Checking header FWCore/Framework/interface/ESProxyFactoryProducer.h
src/FWCore/Framework/interface/ESConsumesCollector.h: In member function 'auto edm::ESConsumesCollector::consumesFrom(const edm::ESInputTag&)':
src/FWCore/Framework/interface/ESConsumesCollector.h:63:94: error: no matching function for call to 'get<1>(__gnu_cxx::__alloc_traits<std::allocator<std::tuple<edm::eventsetup::EventSetupRecordKey, edm::eventsetup::DataKey, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::value_type&)'
       return ESGetToken<Product,Record>{m_transitionID, index, std::get<1>(m_consumer->back()).name().value()};
                                                                                              ^
In file included from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/typelookup.h:45:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/HCTypeTag.h:25,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/EventSetupRecordKey.h:24,
                 from src/FWCore/Framework/interface/ESConsumesCollector.h:31:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:225:5: note: candidate: template<long unsigned int _Int, class _Tp1, class _Tp2> constexpr typename std::tuple_element<_Int, std::pair<_Tp1, _Tp2> >::type& std::get(std::pair<_Tp1, _Tp2>&)
     get(std::pair<_Tp1, _Tp2>& __in) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:225:5: note:   template argument deduction/substitution failed:
src/FWCore/Framework/interface/ESConsumesCollector.h:63:94: note:   '__gnu_cxx::__alloc_traits<std::allocator<std::tuple<edm::eventsetup::EventSetupRecordKey, edm::eventsetup::DataKey, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::value_type {aka std::tuple<edm::eventsetup::EventSetupRecordKey, edm::eventsetup::DataKey, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >}' is not derived from 'std::pair<_Tp1, _Tp2>'
       return ESGetToken<Product,Record>{m_transitionID, index, std::get<1>(m_consumer->back()).name().value()};
                                                                                              ^
In file included from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/typelookup.h:45:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/HCTypeTag.h:25,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/EventSetupRecordKey.h:24,
                 from src/FWCore/Framework/interface/ESConsumesCollector.h:31:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:230:5: note: candidate: template<long unsigned int _Int, class _Tp1, class _Tp2> constexpr typename std::tuple_element<_Int, std::pair<_Tp1, _Tp2> >::type&& std::get(std::pair<_Tp1, _Tp2>&&)
     get(std::pair<_Tp1, _Tp2>&& __in) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:230:5: note:   template argument deduction/substitution failed:
src/FWCore/Framework/interface/ESConsumesCollector.h:63:94: note:   '__gnu_cxx::__alloc_traits<std::allocator<std::tuple<edm::eventsetup::EventSetupRecordKey, edm::eventsetup::DataKey, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::value_type {aka std::tuple<edm::eventsetup::EventSetupRecordKey, edm::eventsetup::DataKey, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >}' is not derived from 'std::pair<_Tp1, _Tp2>'
       return ESGetToken<Product,Record>{m_transitionID, index, std::get<1>(m_consumer->back()).name().value()};
                                                                                              ^
In file included from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/typelookup.h:45:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/HCTypeTag.h:25,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/EventSetupRecordKey.h:24,
                 from src/FWCore/Framework/interface/ESConsumesCollector.h:31:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:235:5: note: candidate: template<long unsigned int _Int, class _Tp1, class _Tp2> constexpr const typename std::tuple_element<_Int, std::pair<_Tp1, _Tp2> >::type& std::get(const std::pair<_Tp1, _Tp2>&)
     get(const std::pair<_Tp1, _Tp2>& __in) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:235:5: note:   template argument deduction/substitution failed:
src/FWCore/Framework/interface/ESConsumesCollector.h:63:94: note:   '__gnu_cxx::__alloc_traits<std::allocator<std::tuple<edm::eventsetup::EventSetupRecordKey, edm::eventsetup::DataKey, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >::value_type {aka std::tuple<edm::eventsetup::EventSetupRecordKey, edm::eventsetup::DataKey, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >}' is not derived from 'const std::pair<_Tp1, _Tp2>'
       return ESGetToken<Product,Record>{m_transitionID, index, std::get<1>(m_consumer->back()).name().value()};
                                                                                              ^
In file included from /cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/typelookup.h:45:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/HCTypeTag.h:25,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/EventSetupRecordKey.h:24,
                 from src/FWCore/Framework/interface/ESConsumesCollector.h:31:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:244:5: note: candidate: template<class _Tp, class _Up> constexpr _Tp& std::get(std::pair<_T1, _T2>&)
     get(pair<_Tp, _Up>& __p) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:244:5: note:   template argument deduction/substitution failed:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:249:5: note: candidate: template<class _Tp, class _Up> constexpr const _Tp& std::get(const std::pair<_T1, _T2>&)
     get(const pair<_Tp, _Up>& __p) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:249:5: note:   template argument deduction/substitution failed:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:254:5: note: candidate: template<class _Tp, class _Up> constexpr _Tp&& std::get(std::pair<_T1, _T2>&&)
     get(pair<_Tp, _Up>&& __p) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:254:5: note:   template argument deduction/substitution failed:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:259:5: note: candidate: template<class _Tp, class _Up> constexpr _Tp& std::get(std::pair<_Up, _Tp>&)
     get(pair<_Up, _Tp>& __p) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:259:5: note:   template argument deduction/substitution failed:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:264:5: note: candidate: template<class _Tp, class _Up> constexpr const _Tp& std::get(const std::pair<_Up, _Tp>&)
     get(const pair<_Up, _Tp>& __p) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:264:5: note:   template argument deduction/substitution failed:
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:269:5: note: candidate: template<class _Tp, class _Up> constexpr _Tp&& std::get(std::pair<_Up, _Tp>&&)
     get(pair<_Up, _Tp>&& __p) noexcept
     ^~~
/cvmfs/cms-ib.cern.ch/nweek-02573/slc7_amd64_gcc700/external/gcc/7.0.0-pafccj/include/c++/7.4.1/utility:269:5: note:   template argument deduction/substitution failed:
gmake: *** [config/SCRAM/GMake/Makefile.chk_headers:44: tmp/slc7_amd64_gcc700/check_header/FWCore/Framework/interface/ESConsumesCollector.h] Error 1
>> Checking header FWCore/Framework/interface/stream/ThinningProducer.h
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h: In member function 'void edm::stream::EDAnalyzerAdaptor<T>::doBeginRun(const edm::RunPrincipal&, const edm::EventSetupImpl&, const edm::ModuleCallingContext*)':
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:132:29: warning: invalid use of incomplete type 'const class edm::EDConsumerBase'
             this->consumer()->esGetTokenIndices(Transition::BeginRun)};
                             ^~
In file included from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/PrincipalGetAdapter.h:104:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/Run.h:21,
                 from src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:24:
/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/EDGetToken.h:31:9: note: forward declaration of 'class edm::EDConsumerBase'
   class EDConsumerBase;
         ^~~~~~~~~~~~~~
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h: In member function 'void edm::stream::EDAnalyzerAdaptor<T>::doEndRun(const edm::RunPrincipal&, const edm::EventSetupImpl&, const edm::ModuleCallingContext*)':
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:150:29: warning: invalid use of incomplete type 'const class edm::EDConsumerBase'
             this->consumer()->esGetTokenIndices(Transition::EndRun)
                             ^~
In file included from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/PrincipalGetAdapter.h:104:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/Run.h:21,
                 from src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:24:
/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/EDGetToken.h:31:9: note: forward declaration of 'class edm::EDConsumerBase'
   class EDConsumerBase;
         ^~~~~~~~~~~~~~
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h: In member function 'void edm::stream::EDAnalyzerAdaptor<T>::doBeginLuminosityBlock(const edm::LuminosityBlockPrincipal&, const edm::EventSetupImpl&, const edm::ModuleCallingContext*)':
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:169:29: warning: invalid use of incomplete type 'const class edm::EDConsumerBase'
             this->consumer()->esGetTokenIndices(Transition::BeginLuminosityBlock)
                             ^~
In file included from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/PrincipalGetAdapter.h:104:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/Run.h:21,
                 from src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:24:
/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/EDGetToken.h:31:9: note: forward declaration of 'class edm::EDConsumerBase'
   class EDConsumerBase;
         ^~~~~~~~~~~~~~
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h: In member function 'void edm::stream::EDAnalyzerAdaptor<T>::doEndLuminosityBlock(const edm::LuminosityBlockPrincipal&, const edm::EventSetupImpl&, const edm::ModuleCallingContext*)':
src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:189:29: warning: invalid use of incomplete type 'const class edm::EDConsumerBase'
             this->consumer()->esGetTokenIndices(Transition::EndLuminosityBlock)
                             ^~
In file included from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/PrincipalGetAdapter.h:104:0,
                 from /build/cmsbld/jenkins/workspace/ib-any-integration/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Framework/interface/Run.h:21,
                 from src/FWCore/Framework/interface/stream/EDAnalyzerAdaptor.h:24:
/cvmfs/cms-ib.cern.ch/week1/slc7_amd64_gcc700/cms/cmssw/CMSSW_10_6_X_2019-04-26-1100/src/FWCore/Utilities/interface/EDGetToken.h:31:9: note: forward declaration of 'class edm::EDConsumerBase'
   class EDConsumerBase;
         ^~~~~~~~~~~~~~

@fabiocos
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit e94318b into cms-sw:master Apr 27, 2019
@wddgit wddgit deleted the bugFixConsumes106X branch August 9, 2019 18:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants