RepeatingCachedRootSource #33332

Dr15Jones · 2021-04-04T15:53:20Z

PR description:

Created the RepeatingCachedRootSource which allows performance tests which are not dependent upon the read speed of the storage medium.

PR validation:

The added unit test passes.

cmsbuild · 2021-04-04T16:01:52Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33332/21883

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File FWCore/Framework/src/ProductResolvers.cc modified in PR(s): Use ServiceWeakToken with asynchronous tasks #33330

cmsbuild · 2021-04-04T16:02:12Z

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

FWCore/Framework
FWCore/TFWLiteSelector
IOPool/Input

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @wddgit this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

Dr15Jones · 2021-04-04T16:04:56Z

please test

cmsbuild · 2021-04-04T18:40:03Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5c5d34/13992/summary.html
COMMIT: f213f96
CMSSW: CMSSW_11_3_X_2021-04-04-0000/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33332/13992/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2640868
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2640845
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 36 files compared)
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

IOPool/Input/src/RepeatingCachedRootSource.cc

makortel · 2021-04-05T16:19:15Z

IOPool/Input/src/RepeatingCachedRootSource.cc

+      std::shared_ptr<edm::WrapperBase> getProduct_(edm::BranchID const& k, edm::EDProductGetter const* ep) final {
+        return m_source->getProduct(m_streamIndex, k, ep);
+      }
+      void mergeReaders_(edm::DelayedReader*) final { assert(false); }


Will this function be ever called?

This DelayedReader is only used on EventPrincipals and events do not do mergers. Therefore it should never be called.

cmsbuild · 2021-04-05T17:36:02Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33332/21893

ERROR: Build errors found during clang-tidy run.

--
FWCore/Framework/src/Worker.h:975:9: error: unknown type name 'ServiceWeakToken'; did you mean 'ServiceToken'? [clang-diagnostic-error]
        ServiceWeakToken weakToken = token;
        ^~~~~~~~~~~~~~~~
        ServiceToken
--
FWCore/Framework/src/Worker.h:979:56: error: no member named 'lock' in 'edm::ServiceToken' [clang-diagnostic-error]
              ServiceRegistry::Operate guard(weakToken.lock());
                                                       ^
FWCore/Framework/src/Worker.h:981:42: error: no member named 'lock' in 'edm::ServiceToken' [clang-diagnostic-error]
                               weakToken.lock(),
                                         ^
FWCore/Framework/src/Worker.h:1046:7: error: unknown type name 'ServiceWeakToken'; did you mean 'ServiceToken'? [clang-diagnostic-error]
      ServiceWeakToken weakToken = serviceToken;
      ^~~~~~~~~~~~~~~~
      ServiceToken
--
FWCore/Framework/src/Worker.h:1052:52: error: no member named 'lock' in 'edm::ServiceToken' [clang-diagnostic-error]
          ServiceRegistry::Operate guard(weakToken.lock());
                                                   ^
Suppressed 930 warnings (928 in non-user code, 2 with check filters).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:128: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

cmsbuild · 2021-04-06T02:45:08Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33332/21896

ERROR: Build errors found during clang-tidy run.

--
FWCore/Framework/src/Worker.h:975:9: error: unknown type name 'ServiceWeakToken'; did you mean 'ServiceToken'? [clang-diagnostic-error]
        ServiceWeakToken weakToken = token;
        ^~~~~~~~~~~~~~~~
        ServiceToken
--
FWCore/Framework/src/Worker.h:979:56: error: no member named 'lock' in 'edm::ServiceToken' [clang-diagnostic-error]
              ServiceRegistry::Operate guard(weakToken.lock());
                                                       ^
FWCore/Framework/src/Worker.h:981:42: error: no member named 'lock' in 'edm::ServiceToken' [clang-diagnostic-error]
                               weakToken.lock(),
                                         ^
FWCore/Framework/src/Worker.h:1046:7: error: unknown type name 'ServiceWeakToken'; did you mean 'ServiceToken'? [clang-diagnostic-error]
      ServiceWeakToken weakToken = serviceToken;
      ^~~~~~~~~~~~~~~~
      ServiceToken
--
FWCore/Framework/src/Worker.h:1052:52: error: no member named 'lock' in 'edm::ServiceToken' [clang-diagnostic-error]
          ServiceRegistry::Operate guard(weakToken.lock());
                                                   ^
Suppressed 930 warnings (928 in non-user code, 2 with check filters).
--
gmake: *** [config/SCRAM/GMake/Makefile.coderules:128: code-checks] Error 2
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 2

Dr15Jones · 2021-04-06T12:42:57Z

code-checks

cmsbuild · 2021-04-06T12:47:58Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33332/21909

This PR adds an extra 52KB to repository

cmsbuild · 2021-04-06T12:48:48Z

Pull request #33332 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

Dr15Jones · 2021-04-06T12:59:50Z

please test

cmsbuild · 2021-04-06T15:44:16Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5c5d34/14019/summary.html
COMMIT: 8ef445b
CMSSW: CMSSW_11_3_X_2021-04-05-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/33332/14019/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 8 differences found in the comparisons
DQMHistoTests: Total files compared: 37
DQMHistoTests: Total histograms compared: 2640868
DQMHistoTests: Total failures: 12
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2640833
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 36 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 155 log files, 37 edm output root files, 37 DQM output files
TriggerResults: no differences found

makortel · 2021-04-06T18:52:05Z

+1

cmsbuild · 2021-04-06T18:52:30Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

makortel · 2021-04-06T18:54:49Z

@fwyzard This might be of interest to you as well (I recall you were asking about a Source that would do the real I/O prior to event loop and recycle the Events).

fwyzard · 2021-04-07T02:22:44Z

Thanks for the mention!
Is the decompression cached as well ?

silviodonato · 2021-04-07T07:18:35Z

+1

Dr15Jones · 2021-04-07T11:42:04Z

Is the decompression cached as well ?

Yes

Dr15Jones added 4 commits April 2, 2021 15:47

Check for end of file earlier

cb8f616

Made DelayedReader return a std::shared_ptr

ca24207

Added RepeatingCachedRootSource

ec4b8af

Add unit test for RepeatingCachedRootSource

f213f96

cmsbuild added this to the CMSSW_11_3_X milestone Apr 4, 2021

cmsbuild added code-checks-pending core-pending orp-pending pending-signatures tests-pending labels Apr 4, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 4, 2021

cmsbuild added tests-started and removed tests-pending labels Apr 4, 2021

cmsbuild added tests-approved and removed tests-started labels Apr 4, 2021

cmsbuild mentioned this pull request Apr 5, 2021

Addition of the MTD to the tracker Navigation #33272

Merged

makortel reviewed Apr 5, 2021

View reviewed changes

Code cleanups

8ef445b

cmsbuild added code-checks-pending tests-pending and removed code-checks-approved tests-approved labels Apr 5, 2021

cmsbuild added code-checks-rejected and removed code-checks-pending labels Apr 5, 2021

cmsbuild added code-checks-rejected and removed code-checks-pending labels Apr 6, 2021

cmsbuild added code-checks-pending and removed code-checks-rejected labels Apr 6, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 6, 2021

cmsbuild added tests-started and removed tests-pending labels Apr 6, 2021

cmsbuild added tests-approved and removed tests-started labels Apr 6, 2021

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels Apr 6, 2021

cmsbuild added orp-approved and removed orp-pending labels Apr 7, 2021

cmsbuild merged commit e4e9af0 into cms-sw:master Apr 7, 2021

Dr15Jones deleted the RepeatingCachedRootSource branch April 7, 2021 11:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RepeatingCachedRootSource #33332

RepeatingCachedRootSource #33332

Dr15Jones commented Apr 4, 2021

cmsbuild commented Apr 4, 2021

cmsbuild commented Apr 4, 2021

Dr15Jones commented Apr 4, 2021

cmsbuild commented Apr 4, 2021

makortel Apr 5, 2021

Dr15Jones Apr 5, 2021

cmsbuild commented Apr 5, 2021

cmsbuild commented Apr 6, 2021

Dr15Jones commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

Dr15Jones commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

makortel commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

makortel commented Apr 6, 2021

fwyzard commented Apr 7, 2021

silviodonato commented Apr 7, 2021

Dr15Jones commented Apr 7, 2021

RepeatingCachedRootSource #33332

RepeatingCachedRootSource #33332

Conversation

Dr15Jones commented Apr 4, 2021

PR description:

PR validation:

cmsbuild commented Apr 4, 2021

cmsbuild commented Apr 4, 2021

Dr15Jones commented Apr 4, 2021

cmsbuild commented Apr 4, 2021

Comparison Summary

makortel Apr 5, 2021

Choose a reason for hiding this comment

Dr15Jones Apr 5, 2021

Choose a reason for hiding this comment

cmsbuild commented Apr 5, 2021

cmsbuild commented Apr 6, 2021

Dr15Jones commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

Dr15Jones commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

Comparison Summary

makortel commented Apr 6, 2021

cmsbuild commented Apr 6, 2021

makortel commented Apr 6, 2021

fwyzard commented Apr 7, 2021

silviodonato commented Apr 7, 2021

Dr15Jones commented Apr 7, 2021