Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DQM: Fix EDMtoMEConverter. #28934

Merged
merged 1 commit into from
Feb 28, 2020
Merged

DQM: Fix EDMtoMEConverter. #28934

merged 1 commit into from
Feb 28, 2020

Conversation

schneiml
Copy link
Contributor

PR description:

This PR fixes a problem pointed out by @mmusich related to the AlCa multi-run harvesting workflows, caused by #28622.

In the past, the DQMStore, when used with "collate" mode, would
automatically merge the TH1 object passed into a book*() call with an
existing object. However, the merge logic in the DQMStore is rather
conservative and didn't merge objects with SetCanExtend set. So, to
work around this, EDMtoMEConverter would merge these objects itself.

The new DQMStore does not have the "collate" mode any more and all the
merging has to happen in the input modules. This fixes
EDMtoMEConverter to always use its local merge logic.

Arguably the ME merge logic should be factored out of DQMRootSource and
EDMtoMEConverter and shared between the two, but for now this small fix
should work to get the AlCa workflows working again.

PR validation:

Privately provided test configuration seems to provide correct results, needs further testing by AlCa.

I have not much confidence that the EDMtoMEConverter behaves correctly for all combinations of settings, but in the end this module is there to support the AlCa workflows dating back to Run1 and the only thing that matters is that these work correctly.

In the past, the DQMStore, when used with "collate" mode, would
automatically merge the `TH1` object passed into a `book*()` call with an
existing object. However, the merge logic in the `DQMStore` is rather
conservative and didn't merge objects with `SetCanExtend` set. So, to
work around this, `EDMtoMEConverter` would merge these objects itself.

The new `DQMStore` does not have the "collate" mode any more and all the
merging has to happen in the input modules. This fixes
`EDMtoMEConverter` to *always* use its local merge logic.

Arguably the ME merge logic should be factored out of DQMRootSource and
EDMtoMEConverter and shared between the two, but for now this small fix
should work to get the AlCa workflows working again.
@cmsbuild
Copy link
Contributor

The code-checks are being triggered in jenkins.

@cmsbuild
Copy link
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28934/13736

  • This PR adds an extra 16KB to repository

@cmsbuild
Copy link
Contributor

A new Pull Request was created by @schneiml (Marcel Schneider) for master.

It involves the following packages:

DQMServices/Components

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@barvic this is something you requested to watch as well.
@davidlange6, @silviodonato, @fabiocos you are the release manager for this.

cms-bot commands are listed here

@schneiml
Copy link
Contributor Author

please test

Wondering about the unit tests and wf 1001, otherwise this code should not affect anything.

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 12, 2020

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4621/console Started: 2020/02/12 15:22

@mmusich
Copy link
Contributor

mmusich commented Feb 12, 2020

thanks @schneiml !
@silviodonato I am wondering if there is a way to run the tests with higher statistics in the PR tests.

@silviodonato
Copy link
Contributor

@silviodonato I am wondering if there is a way to run the tests with higher statistics in the PR tests.

I don't think it is possible. @smuzaffar ? @fabiocos ?

@smuzaffar
Copy link
Contributor

currently not possible

@cmsbuild
Copy link
Contributor

+1
Tested at: 2972d1d
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ac6b9c/4621/summary.html
CMSSW: CMSSW_11_1_X_2020-02-12-1100
SCRAM_ARCH: slc7_amd64_gcc820

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ac6b9c/4621/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 34
  • DQMHistoTests: Total histograms compared: 2694005
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2693658
  • DQMHistoTests: Total skipped: 346
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
  • Checked 147 log files, 16 edm output root files, 34 DQM output files

@mmusich
Copy link
Contributor

mmusich commented Feb 12, 2020

For some reason relmon for wf 1001.0 is not available... https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_11_1_X_2020-02-12-1100+ac6b9c/35044/

looking to other PRs seems a persistent issue in PR tests. Who's responsible to fix that? PdmV?

@schneiml
Copy link
Contributor Author

@tocheng @mmusich please let me know if this properly fixes the EDMtoME related problems. I'd sign it then.

@mmusich
Copy link
Contributor

mmusich commented Feb 17, 2020

@schneiml, I'll hopefully get to this later this week.

@cmsbuild
Copy link
Contributor

New categories assigned: alca

@christopheralanwest,@franzoni,@tlampen,@pohsun,@tocheng you have been requested to review this Pull request/Issue and eventually sign? Thanks

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 19, 2020

The tests are being triggered in jenkins.
Tested with other pull request(s) #28994
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4788/console Started: 2020/02/19 20:29

@cmsbuild
Copy link
Contributor

+1
Tested at: 2972d1d
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dea72c/4788/summary.html
CMSSW: CMSSW_11_1_X_2020-02-19-1100
SCRAM_ARCH: slc7_amd64_gcc820

@cmsbuild
Copy link
Contributor

Comparison job queued.

@cmsbuild
Copy link
Contributor

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-dea72c/4788/summary.html

Comparison Summary:

  • No significant changes to the logs found
  • Reco comparison results: 4 differences found in the comparisons
  • DQMHistoTests: Total files compared: 34
  • DQMHistoTests: Total histograms compared: 2694086
  • DQMHistoTests: Total failures: 2
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 2693765
  • DQMHistoTests: Total skipped: 319
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.063 KiB( 33 files compared)
  • DQMHistoSizes: changed ( 1001.0 ): 0.063 KiB AlCaReco/SiStripGainsAAG
  • Checked 147 log files, 16 edm output root files, 34 DQM output files

@silviodonato
Copy link
Contributor

@tocheng @christopheralanwest @tlampen do you have any comments on this PR?

@schneiml
Copy link
Contributor Author

We should get this merged at some point -- without this PR, the AlCa jobs are broken. 11_1_0_pre3 is broken in that respect and we should certainly have a fix for pre4.

@mmusich
Copy link
Contributor

mmusich commented Feb 27, 2020

I fully support Marcel here!

@pohsun
Copy link

pohsun commented Feb 27, 2020

+1

@cmsbuild
Copy link
Contributor

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

@silviodonato
Copy link
Contributor

+1

@cmsbuild cmsbuild merged commit 804ccf4 into cms-sw:master Feb 28, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants