Making the boosted double SV tagger thread-safe

[ferencek]

fastjet::contrib::Njettiness is not thread-safe and therefore should not be used as a data member of an ESProducer. This PR addresses the problem by defining a local instance of fastjet::contrib::Njettiness instead of using a global instance defined as a data member of the CandidateBoostedDoubleSecondaryVertexComputer class.

This PR is urgent since it fixes CMSSW crashes occurring in the multi-threaded jobs.

[cmsbuild]

A new Pull Request was created by @ferencek (Dinko Ferenček) for CMSSW_8_0_X.

It involves the following packages:

RecoBTag/SecondaryVertex

@cmsbuild, @cvuosalo, @davidlange6, @slava77 can you please review it and eventually sign? Thanks.
@ferencek, @acaudron, @pvmulder, @imarches this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

Following commands in first line of a comment are recognized

• +1|approve[d]|sign[ed]: L1/L2's to approve it
• -1|reject[ed]: L1/L2's to reject it
• assign <category>[,<category>[,...]]: L1/L2's to request signatures from other categories
• unassign <category>[,<category>[,...]]: L1/L2's to remove signatures from other categories
• hold: L1/all L2's/release manager to mark it as on hold
• unhold: L1/user who put this PR on hold
• merge: L1/release managers to merge this request
• [@cmsbuild,] please test: L1/L2 and selected users to start jenkins tests
• [@cmsbuild,] please test with cms-sw/cmsdist#<PR>: L1/L2 and selected users to start jenkins tests using externals from cmsdist

[slava77]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/10166/console

[cmsbuild]

+1
Tested at: b7970d9
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12680/10166/summary.html

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-12680/10166/summary.html

[Dr15Jones]

Just to give some sense of urgency, over 120 workflows are failing in CMSSW_8_0_THREADED_X without the thread-safe fix.

[smuzaffar]

I am approving it and starting 00h00 IB now

[smuzaffar]

merge

[cvuosalo]

+1

For #12680 b7970d9

Urgent bug fix for to prevent multi-threaded jobs from crashing. There should be no change in monitored quantities. #12679 is the 76X version of this PR.

The code change is satisfactory, and Jenkins tests against baseline CMSSW_8_0_X_2015-12-04-1100 show no significant differences, as expected. An extended test of workflow 25202.0_TTbar_13 with 70 events against baseline CMSSW_8_0_X_2015-12-04-1100 also shows no significant differences. Timing measurements show no significant difference. A related module even seems to show a tiny timing decrease:

  delta/mean delta/orJob     original                   new       module name
  ---------- ------------     --------                  ----       ------------
   -0.066339      -0.00%        15.36 ms/ev ->        14.37 ms/ev pfBoostedDoubleSecondaryVertexAK8BJetTagsAK8

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_8_0_X IBs (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @Degano, @smuzaffar

[smuzaffar]

@Dr15Jones , we now have a new issue in threaded IBs.
cmsRun: Nsubjettiness/MeasureDefinition.cc:168: fastjet::contrib::TauPartition fastjet::contrib::MeasureDefinition::get_partition(const std::vector&, const std::vector&) const: Assertion `has_beam()' failed.

[ferencek]

@smuzaffar, can you give an example of a workflow that is failing?

[smuzaffar]

@ferencek , see the workflows here https://cms-sw.github.io/relvalLogDetail.html#slc6_amd64_gcc493;CMSSW_8_0_THREADED_X_2015-12-09-1100

for example 4.67, 4.68 (failing in step3)

[Dr15Jones]

I think the problem is fastjet::contrib::DefaultMeasure::UpdateAxesFast

https://github.com/cms-externals/fastjet-contrib/blob/cms/v1.020/Nsubjettiness/MeasureDefinition.cc#L387

[cvuosalo]

This error is already present in IBs before this PR was merged, for example CMSSW_8_0_THREADED_X_2015-12-04-1100:

https://cmssdt.cern.ch/SDT/cgi-bin/buildlogs/slc6_amd64_gcc493/CMSSW_8_0_THREADED_X_2015-12-04-1100/pyRelValMatrixLogs/run/4.25_WElSkim2011A+WElSkim2011A+HLTDSKIM+RECODreHLT+HARVESTDreHLT/step3_WElSkim2011A+WElSkim2011A+HLTDSKIM+RECODreHLT+HARVESTDreHLT.log

[Dr15Jones]

I think I have the needed fix to fastjet-contrib

cms-externals/fastjet-contrib#9

[ferencek]

@Dr15Jones, thanks for the fix. I wonder why this issue did not show up earlier since there are other places in CMSSW where Njettiness is used, in particular in the MiniAOD production. Also, what is the mechanism to propagate this fix upstream.