Faster provenance handling

[Dr15Jones]

PR description:

The ProductProvenanceRetriever used to hold the parentage information for the Event data products now uses a faster data structure and avoids resetting itself every Event.

PR validation:

The framework unit tests all pass.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28536/13009

• This PR adds an extra 60KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File FWCore/Framework/interface/EventPrincipal.h modified in PR(s): Faster setup of EventPrincipal #28505
  • File FWCore/Framework/interface/Principal.h modified in PR(s): Faster setup of EventPrincipal #28505
  • File FWCore/Framework/src/EventPrincipal.cc modified in PR(s): Faster setup of EventPrincipal #28505
  • File FWCore/Framework/src/Principal.cc modified in PR(s): Faster setup of EventPrincipal #28505
  • File IOPool/Streamer/src/StreamerInputSource.cc modified in PR(s): Faster setup of EventPrincipal #28505

[cmsbuild]

A new Pull Request was created by @Dr15Jones (Chris Jones) for master.

It involves the following packages:

DataFormats/Provenance
DataFormats/Streamer
FWCore/Framework
IOPool/Streamer

@cmsbuild, @smuzaffar, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @rovere, @wddgit this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[Dr15Jones]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3778/console Started: 2019/12/04 00:20

[cmsbuild]

-1

Tested at: a9e7916

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e3a77e/3778/summary.html

I found follow errors while testing this PR

Failed tests: ClangBuild

• Clang:

I found compilation warning while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' scram build -k -j 32 COMPILER='llvm compile'

See details on the summary page.

[cmsbuild]

Comparison not run due to Build errors/Fireworks only changes/No short matrix requested (RelVals and Igprof tests were also skipped)

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3798/console Started: 2019/12/04 16:56

[cmsbuild]

+1
Tested at: 7adf9c9
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e3a77e/3798/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

• A fix for importing dependencies in DQM bin by bin tool #28494 @andrius-k: A fix for importing dependencies in DQM bin by bin tool
• Update truncated tube shape parameters to match new version of DD4hep #28496 @cvuosalo: Update truncated tube shape parameters to match new version of DD4hep
• Adding HTXS Stage 1.2 classification #28444 @adewit: Adding HTXS Stage 1.2 classification
• DQM: Script to locally run the bin-to-bin comparison tool #28501 @schneiml: DQM: Script to locally run the bin-to-bin comparison tool
• make HB TDC conversion configurable #28474 @lwang046: make HB TDC conversion configurable

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e3a77e/3798/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e3a77e/3798/git-merge-result

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e3a77e/3798/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2793840
• DQMHistoTests: Total failures: 26
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2793473
• DQMHistoTests: Total skipped: 341
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 147 log files, 16 edm output root files, 34 DQM output files

[Dr15Jones]

+1
The comparison failures are the same as seen in #28474 which was included by the bot in the testing.

[fabiocos]

@smuzaffar here I see another case of tests run against 11_0_X for a 11_1_X PR

[fabiocos]

@qliphy @agrohsje this looks a technical update, I am performing a quick cross check on LHE provenance, please have a look, i will integrate this PR ass soon as I have finished

[smuzaffar]

correct, This issue has been identified and fixed. The ib baseline generation job was using your old branch for 11.0.X [a]. This was done to test your changes but it was never reverted. This only effected the generation of baseline job to use an old cms-bot. Note that https://github.com/cms-sw/cms-bot/blob/master/run-ib-comparison-baseline in your branch and cms-bot master is identical, so this did not break any thing till now.

Problem is that in your branch has DEV release set to CMSSW_11_0_X while now we move to CMSSW_11_1_X. Anyway, feel free to re-tests if case you want to get clean

[a]

if [ $(echo $RELEASE_FORMAT | grep 'CMSSW_11_0_X_' | wc -l) -gt 0 ] ; then
  git clone -b fc-IBperf https://github.com/fabiocos/cms-bot
else
  git clone https://github.com/cms-sw/cms-bot
fi

[fabiocos]

@smuzaffar thanks, good that it is understood, I'll check and clean my branch

[fabiocos]

@qliphy @agrohsje please check, I will integrate this PR in next IB

[qliphy]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1