Fix Relval steps for GEN workflow 535

[SanghyunKo]

Currently, GEN Relval workflow 535 produces corrupted DQM files, which makes RelMon comparison plot unavailable. For this GEN-only Relval workflow, SIM step is not needed and should be replaced by VALIDATION:genvalid step like other GEN-only workflows to make Relval works (see discussions in PR #23816 ).

Also, externalLHEProducer of the workflow 535 makes use of exactly same TTbar Powheg gridpack, hence made it imported for maintainability.

Fix of PR #23802.

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24833/6783

[cmsbuild]

A new Pull Request was created by @SanghyunKo (Sanghyun Ko) for master.

It involves the following packages:

Configuration/Generator
Configuration/PyReleaseValidation

@cmsbuild, @efeyazgan, @zhenhu, @perrozzi, @prebello, @kpedro88, @pgunnell, @alberto-sanchez, @qliphy can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @makortel this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[zhenhu]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30935/console Started: 2018/10/09 07:40

[cmsbuild]

+1
Tested at: a78c7d3
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24833/30935/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24833/30935/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 2 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3162979
• DQMHistoTests: Total failures: 2
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3162780
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 134 log files, 14 edm output root files, 32 DQM output files

[kpedro88]

+upgrade

[zhenhu]

please test workflow 535

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30947/console Started: 2018/10/09 18:33

[cmsbuild]

+1
Tested at: a78c7d3
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24833/30947/summary.html

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24833/30947/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-24833/535.0_TTbar_13TeV_Pow_herwig7+TTbar_13TeV_Pow_herwig7+HARVESTGEN

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 32
• DQMHistoTests: Total histograms compared: 3162979
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3162781
• DQMHistoTests: Total skipped: 197
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
• Checked 134 log files, 14 edm output root files, 32 DQM output files

[zhenhu]

+1

[fabiocos]

@SanghyunKo @alberto-sanchez the workflow runs, but in the step1 log file I see:

• A warning exception occurred in the initialization of EventGenerator:
  LesHouchesReader LesHouchesReader has the IDWTUP flag set to -4 which is not supported by this reader, the produced events may not be sampled correctly. It is up to sub-classes of LesHouchesReader to correctly convert to match IDWTUP +/- 1. Will try to make intelligent guesses to get correct statistics.
  In most cases this should be sufficient. Unset WeightWarnings to avoid this message,or set Weighted to on.
• A warning exception occurred in the initialization of EventGenerator:
  LesHouchesReader LesHouchesReader has the IDWTUP flag set to -4 which is not supported by this reader, the produced events may not be sampled correctly. It is up to sub-classes of LesHouchesReader to correctly convert to match IDWTUP +/- 1. Will try to make intelligent guesses to get correct statistics.
  In most cases this should be sufficient. Unset WeightWarnings to avoid this message,or set Weighted to on.
  Error: The object '/Herwig/EventHandlers/Luminosity' was not created as another object with that name already exists.
  Error: The object '/Herwig/EventHandlers/LesHouchesHandler' was not created as another object with that name already exists.
  Error: The object '/Herwig/Cuts/NoCuts' was not created as another object with that name already exists.
  Error: The object '/Herwig/Partons/LHAPDF' was not created as another object with that name already exists.
  Error: The object '/Herwig/EventHandlers/LesHouchesReader' was not created as another object with that name already exists.
  Error: The object '/Herwig/Partons/PDFSet' was not created as another object with that name already exists.

This error messages do not look specifically related to this PR, I would say, but I think they should be understood and possibly addressed. Is the DQM out finally usable?

[SanghyunKo]

Hi @fabiocos , actually the same question was raised in PR #23732, and you can see the answer of the original author (@Andrej-CMS ) at here.

For the DQM files, I tested the full Relval process and generated comparison plots for the workflow 535. You can check the EDM files & logs at '/afs/cern.ch/work/s/sako/public/Validation/openloops/CMSSW_10_3_0_pre5/src/wf535/step1/01', and the comparison plots at here.

Although this PR does not address errors above, at least generated DQM files should be usable with this fix (previously, errors saying that can't find the histograms occurred during the harvesting step.).

[fabiocos]

@SanghyunKo ok, did you verify that the DQM files were usable?

[SanghyunKo]

@fabiocos Yes, they're usable. The step2 (which uses DQM files as input) now works well, and also comparison plots can be drawn using files produced after the step2 (link).

They can be checked if anyone wants at:
/afs/cern.ch/work/s/sako/public/Validation/openloops/CMSSW_10_3_0_pre5/src/wf535/step2

[fabiocos]

@SanghyunKo thanks, @alberto-sanchez @efeyazgan @qliphy could you please check and sign it in case?

[alberto-sanchez]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[fabiocos]

+1