DQMIO.root not produced in all gen and extendgen workflows

[zhenhu]

For 10_4_0_pre2:
DQMIO.root files were not produced in all gen and extendedgen workflows
503,504,505,506,512,531,541,545
507,508,509,510,513,514,522,523,535,551,552,554,555,556,560,561,562

If there is no problem, the output root files will be automatically uploaded here:
https://cmsweb.cern.ch/dqm/relval/data/browse/ROOT/RelVal/CMSSW_10_4_x/
But we can not find the gen output for 10_4_0_pre2.

[cmsbuild]

A new Issue was created by @zhenhu Zhen Hu.

@davidlange6, @Dr15Jones, @smuzaffar, @fabiocos, @kpedro88 can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[zhenhu]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@andrius-k,@schneiml,@kmaeshima you have been requested to review this Pull request/Issue and eventually sign? Thanks

[zhenhu]

assign generators

[cmsbuild]

New categories assigned: generators

@perrozzi,@alberto-sanchez,@qliphy,@efeyazgan you have been requested to review this Pull request/Issue and eventually sign? Thanks

[zhenhu]

assign pdmv

[cmsbuild]

New categories assigned: pdmv

@prebello,@pgunnell,@zhenhu you have been requested to review this Pull request/Issue and eventually sign? Thanks

[fabiocos]

@zhenhu sorry, if I run a few of these workflows in 10_4_0_pre2 I see them completing smoothly and producing a DQM output file. So why should this be a problem in the release itself?

[zhenhu]

Hi @fabiocos , if it's not a problem in the release, it might be a computing issue.
Actually, this issue happened in ALL generator workflows, but not in any other workflows, which made me think it could be due to some recent changes in gen workflows.

[zhenhu]

A JIRA ticket has been opened for this issue
https://its.cern.ch/jira/browse/CMSCOMPPR-4020

[zhenhu]

After some study, we find this issue is caused by PR #24920
More discussion here:
#24920 (comment)

[fabiocos]

@zhenhu the proposed fix is now in 10_4_0_pre3, please check whether the RelVal production now is ok

[zhenhu]

@fabiocos , the problem has been fixed in 10_4_0_pre3. This issue can be closed now.