Add a mechanism to adjust the pileup distribution in PreMixingModule

[makortel]

This PR adds a mechanism to adjust the pileup distribution of the premixed pileup library in premixing stage2 (pileup overlay). The mechanism (that is off by default) is the following

• For each true pileup value, a probability is defined in the configuration (see example below)
• When PreMixingModule reads an event from the pileup library
  1. read the true pileup value from the pileup event
  2. generate a (0,1) random number
  3. if "number > prob(true pileup)", discard the pileup event and continue on i., otherwise use the pileup event as usual

If a pileup event is encountered with a true pileup value outside of the range in configuration, the probability is considered to be 0. Naively this would lead to an infinite loop on misconfiguration (if the configured pileup range would not cover at all the generated pileup), so a following protection is included: an exception is thrown if N consecutive pileup events landing outside of the configured pileup range are encountered (N is configurable, the exception can also be disabled).

This mechanism required adding a way to skip events in VectorInputSource.

Example configuration

mixData = cms.EDProducer("PreMixingModule",
    ...
    adjustPileupDistribution = cms.VPSet(
        cms.PSet(
            firstRun = cms.uint32(1), # first run of which this snippet is active
            firstBinPileup = cms.uint32(10), # pileup of the first probability bin
            pileupProbabilities = cms.vdouble(0.1, 0.5, 1.0, 0.4) # probabilities for pileups 10,11,12,14
        ),
        cms.PSet(
            firstRun = cms.uint32(2),
            firstBinPileup = cms.uint32(10),
            pileupProbabilities = cms.vdouble(0.2, 0.3, 0.5, 1.0)
        ),
    ),
    ...
)

Tested in 10_4_0, no changes expected.

[makortel]

FYI @Dr15Jones @mdhildreth

[cmsbuild]

The code-checks are being triggered in jenkins.

[makortel]

@cmsbuild, please test

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25770/8189

• This PR adds an extra 60KB to repository

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/32849/console Started: 2019/01/26 00:50
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/32855/console Started: 2019/01/27 15:04

[cmsbuild]

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

DQM/SiStripMonitorHardware
FWCore/Sources
IOPool/SecondaryInput
Mixing/Base
SimGeneral/DataMixingModule
SimGeneral/MixingModule
SimGeneral/PileupInformation
SimGeneral/PreMixingModule

@smuzaffar, @andrius-k, @Dr15Jones, @kmaeshima, @schneiml, @civanch, @mdhildreth, @cmsbuild, @jfernan2 can you please review it and eventually sign? Thanks.
@hdelanno, @wddgit, @Martin-Grunewald, @fioriNTU, @jandrea, @idebruyn, @threus this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[cmsbuild]

+1
Tested at: a48dc6e
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25770/32855/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25770/32855/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3097440
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3097242
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 133 log files, 14 edm output root files, 32 DQM output files

[cmsbuild]

Pull request #25770 was updated. @smuzaffar, @andrius-k, @Dr15Jones, @kmaeshima, @schneiml, @civanch, @mdhildreth, @cmsbuild, @jfernan2 can you please check and sign again.

[Dr15Jones]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/32888/console Started: 2019/01/29 22:01

[cmsbuild]

+1
Tested at: 1311056
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25770/32888/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-25770/32888/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3097440
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3097242
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 133 log files, 14 edm output root files, 32 DQM output files

[civanch]

+1

[andrius-k]

+1

[Dr15Jones]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1

the added unit tests run successfully