Change premixing library format from RAW to DIGI #24184
Conversation
|
The code-checks are being triggered in jenkins. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24184/5873 |
|
A new Pull Request was created by @makortel (Matti Kortelainen) for master. It involves the following packages: Configuration/EventContent @perrotta, @pgunnell, @prebello, @cmsbuild, @thomreis, @zhenhu, @emeschi, @civanch, @mdhildreth, @fabozzi, @nsmith-, @rekovic, @franzoni, @kpedro88, @mommsen, @slava77, @GurpreetSinghChahal, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
makortel
changed the title
|
The tests are being triggered in jenkins. |
|
The premixing library file size stays essentially the same. In the 2018 ttbar+PU, the |
|
hi @makortel I cannot access your links above (private DQM GUI) |
|
@prebello See the SSH tunnel recipe or use the links below that work only from CERN network and 250402.1 (2016 FastSim, added in #24149) |
|
+1 The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic: |
|
Comparison job queued. |
|
+operations the event content is modified for step2 of the test workflow, i.e. premix stage1, without evidence of loss of functionality |
|
+1 |
|
+1 |
|
@makortel as you have warned, a number of premixing test workflows are broken because of this PR, starting from CMSSW_10_3_X_2018-09-25-1100 . In my understanding the fix passes through a regeneration of the input files with the new event content, so it is almost unavoidable to ha period when some IBs are broken, please correct me if I am wrong. On one hand I think that it would be useful to have this feature integrated in 10_3_X for easier tests, although in order to fix the situation and have a complete premixing validation we would need one extra pre-release cycle. @franzoni @prebello @pgunnell @zhenhu could you please clarify whether it is reasonable to produce quickly new input files and have premixing validation in a possible fast 10_3_0_pre6 to be done just for this purpose? |
|
@fabiocos
premixing inputs are regenerated at each prerelease.. no need for a pre6.
… On Sep 26, 2018, at 9:56 AM, Fabio Cossutti ***@***.***> wrote:
@makortel as you have warned, a number of premixing test workflows are broken because of this PR, starting from CMSSW_10_3_X_2018-09-25-1100 . In my understanding the fix passes through a regeneration of the input files with the new event content, so it is almost unavoidable to ha period when some IBs are broken, please correct me if I am wrong. On one hand I think that it would be useful to have this feature integrated in 10_3_X for easier tests, although in order to fix the situation and have a complete premixing validation we would need one extra pre-release cycle.
@franzoni @prebello @pgunnell @zhenhu could you please clarify whether it is reasonable to produce quickly new input files and have premixing validation in a possible fast 10_3_0_pre6 to be done just for this purpose?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
@fabiocos, as @davidlange6 said above, as soon as this PR is merged for the next pre release then the RelValPREMIX... libraries will be regenerated with the new input format. |
|
@prebello @davidlange6 that new input files are produced with pre5 is what I expect, but in principle then PyReleaseValidation should be updated accordingly in places like https://cmssdt.cern.ch/lxr/source/Configuration/PyReleaseValidation/python/relval_steps.py#0612 I understand that this is going to be done manually, providing a PR that can be quickly merged to stop the failures, but without the need of an explicit build to start the production of premixed samples for the PMX campaign. Is this that you mean? |
|
On Sep 26, 2018, at 11:39 AM, Fabio Cossutti ***@***.***> wrote:
@prebello @davidlange6 that new input files are produced with pre5 is what I expect, but in principle then PyReleaseValidation should be updated accordingly in places like
https://cmssdt.cern.ch/lxr/source/Configuration/PyReleaseValidation/python/relval_steps.py#0612
I understand that this is going to be done manually, providing a PR that can be quickly merged to stop the failures, but without the need of an explicit build to start the production of premixed samples for the PMX campaign. Is this that you mean?
right
…
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.
|
This is correct (and unfortunate, I agree, but I don't know a way around it). |
@makortel me and @davidlange6 have already replied about it. Did you see our comments above?
|
|
@prebello Yes I did. |
The premixing for phase2 is already using (sim) DIGI as the premixing library format, because it allows simplicity (no need to pack+unpack) with only tiny cost in the file size. As outlined in the spring O&C week
https://indico.cern.ch/event/711343/contributions/2964389/attachments/1631761/2601778/slides_premix_20180412.pdf
this PR does the same for the run2 premixing. In addition
Since the packing+unpacking are (or can be) lossy, there will be small changes in run2 FullSim and FastSim. However, the changes should be towards the classical mixing (as the workflow becomes more similar), and this is what I see e.g. for CSC digis.
Here are links (to my private DQM GUI; black is the release and blue this PR) to 100-event comparison of 250202.181 (2018 ttbar+PU FullSim)
http://127.0.0.1:8081/dqm/relval/start?runnr=1;dataset=/RelValTTbar_13/CMSSW_10_2_0-PU50_2018_premixing_100ev_orig_v1-v1/DQMIO;sampletype=offline_relval;filter=all;referencepos=ratiooverlay;referenceshow=all;referencenorm=True;referenceobj1=other%3A%3A/RelValTTbar_13/CMSSW_10_2_0-PU50_2018_premixing_100ev_dev_v1-v1/DQMIO%3A;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=M;root=;focus=;zoom=no;
10-event for 250202.171 (2017)
http://127.0.0.1:8081/dqm/relval/start?runnr=1;dataset=/RelValTTbar_13/CMSSW_10_2_0-PU35_2017_premixing_10ev_orig_v1-v1/DQMIO;sampletype=offline_relval;filter=all;referencepos=ratiooverlay;referenceshow=all;referencenorm=True;referenceobj1=other%3A%3A/RelValTTbar_13/CMSSW_10_2_0-PU35_2017_premixing_10ev_dev_v1-v1/DQMIO%3A;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=M;root=;focus=;zoom=no;
250202.1 (2016)
http://127.0.0.1:8081/dqm/relval/start?runnr=1;dataset=/RelValTTbar_13/CMSSW_10_2_0-PU35_2016_premixing_10ev_orig_v1-v1/DQMIO;sampletype=offline_relval;filter=all;referencepos=ratiooverlay;referenceshow=all;referencenorm=True;referenceobj1=other%3A%3A/RelValTTbar_13/CMSSW_10_2_0-PU35_2016_premixing_10ev_dev_v1-v1/DQMIO%3A;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=M;root=;focus=;zoom=no;
and 250402.1 (2016 FastSim, added in #24149)
http://127.0.0.1:8081/dqm/relval/start?runnr=1;dataset=/RelValTTbar_13/CMSSW_10_2_0-PU35_FS2016_premixing_10ev_orig_v1-v1/DQMIO;sampletype=offline_relval;filter=all;referencepos=ratiooverlay;referenceshow=all;referencenorm=True;referenceobj1=other%3A%3A/RelValTTbar_13/CMSSW_10_2_0-PU35_FS2016_premixing_10ev_dev_v1-v1/DQMIO%3A;referenceobj2=none;referenceobj3=none;referenceobj4=none;search=;striptype=object;stripruns=;stripaxis=run;stripomit=none;workspace=Everything;size=M;root=;focus=;zoom=no;
with the following SSH tunnel recipe
Tested in 10_2_0 (rebased on top of
CMSSW_10_3_X_2018-08-29-2300CMSSW_10_3_X_2018-09-04-1100), expecting small changes in run2 FullSim and FastSim premixing, but no changes in phase2 (FullSim) premixing. Because the premixing library format changes, this PR breaks all stage2-only premixing workflows.@kpedro88 @mdhildreth