wf 8.0 failing in all Jenkins tests

[tvami]

wf 8.0 fails even with the simplest PRs in Jenkins, for example:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-13-1700+7d20f9/44780/validateJR.html

I understand that this is a long standing issue, and I'm wondering how it should be resolved. I cannot reproduce the problem in local testing

cmsrel CMSSW_12_1_X_2021-08-13-1700
cd CMSSW_12_1_X_2021-08-13-1700/src/
cmsenv
runTheMatrix.py -l 8.0 -j 8

leads to
1 1 1 1 1 tests passed, 0 0 0 0 0 failed

Should we maybe just remove this from the Jenkins tests? It seems to be a BeamHalo run1_mc cosmics workflow.

[cmsbuild]

A new Issue was created by @tvami Tamas Vami.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

Just to clarify, the "failure" in PR tests

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-13-1700+7d20f9/44780/validateJR.html

is in the DQM output comparisons, while the runTheMatrix output

1 1 1 1 1 tests passed, 0 0 0 0 0 failed

checks only if the workflow technically runs or not.

[makortel]

Following the link to the failing comparison leads to https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-13-1700+7d20f9/44780/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/Pixel_AdditionalPixelErrors.html

that only says "Skipped: 0.2% (1)" without pointing what exactly was skipped.

[makortel]

assign dqm

[cmsbuild]

New categories assigned: dqm

@jfernan2,@kmaeshima,@rvenditti,@andrius-k,@ErnestaP,@ahmad3213 you have been requested to review this Pull request/Issue and eventually sign? Thanks

[tvami]

Thanks @makortel for the clarification. Indeed following the links I see that
"Failing Comparisons: avgfedDigiOccvsLumi" here
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-13-1700+7d20f9/44780/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/Pixel.html

and "Skiped Comparisons byLumiErrors" here
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-13-1700+7d20f9/44780/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/Pixel_AdditionalPixelErrors.html

[jfernan2]

Is there any way to see from RelMon the log error of those skipped/failing comparisons? The DQM bin by bin tool does not show any problem on those Pixel folders

[smuzaffar]

@jfernan2 , you can go to parent directory ( just remove the file part of the url in your browser e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_12_1_X_2021-08-13-1700+7d20f9/44780/ ) and then look for 8.0 log file.

[jfernan2]

I have manually inspected the 2 root files (baseline and from a PR) for the wf8:

• for the skipped one, averageDigiOccupancy, both baseline and PR file have 80 entries with mean 0 and stddev =0 (overflow and underflow are zero too)

• for the failing one, consequently avgfedDigiOccvsLumi, the first LS is filled with NaN for both baseline and PR DQM root files
  

Given the nature of this wf 8.0 (BeamHalo) this plot has probably no sense

[tvami]

Hi @jfernan2 thanks for looking into this. I'm wondering why the comparison of NaN vs NaN doesnt give an agreement.
Anyway, would you then recommend to remove wf 8.0 from the Jenkins tests?

[davidlange6]

By definition nan!=nan (or anything else) - Better if the code producing the nan would instead produce some valid result.

…

On Aug 16, 2021, at 7:10 PM, Tamas Vami ***@***.***> wrote: Hi @jfernan2 thanks for looking into this. I'm wondering why the comparison of NaN vs NaN doesnt give an agreement. Anyway, would you then recommend to remove wf 8.0 from the Jenkins tests? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[jfernan2]

The code can probably not give valid results in this case since there is no FED info for Pizel.
What about blacklisting them in RelMon? Indeed I see they are in a blacklist:

cmssw/Utilities/RelMon/data/blacklist.txt

Lines 75 to 76 in 6d2f660

	Pixel/averageDigiOccupancy
	Pixel/AdditionalPixelErrors/byLumiErrors

I am adding here Tracker DQM contacts in case they shred some light:
@sroychow @mmusich

[davidlange6]

Its typically straightforward to protect against generating nans.. what's the code that is doing this? (Of course will still be not useful in this workflow but then the histograms will be enabled for other workflows)

…

On Aug 16, 2021, at 7:22 PM, jfernan2 ***@***.***> wrote: The code can probably not give valid results in this case since there is no FED info for Pizel. What about blacklisting them in RelMon? Indeed I see they are in a blacklist: https://github.com/cms-sw/cmssw/blob/6d2f66057131baacc2fcbdd203588c41c885b42c/Utilities/RelMon/data/blacklist.txt#L75-L76 I am adding here Tracker DQM contacts in case they shred some light: @sroychow @mmusich — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.

[jfernan2]

cmssw/DQM/SiPixelMonitorDigi/src/SiPixelDigiSource.cc

Lines 133 to 146 in 639903d

	avgfedDigiOccvsLumi->setBinContent(thisls, i + 1, nDigisPerFed[i]); //Same plot vs lumi section
	}
	if (modOn) {
	if (thisls % 10 == 0)
	averageDigiOccupancy->Fill(
	i,
	averageOcc); // "modOn" basically mean Online DQM, in this case fill histos with actual value of digi fraction per fed for each ten lumisections
	if (avgfedDigiOccvsLumi && thisls % 5 == 0)
	avgfedDigiOccvsLumi->setBinContent(
	int(thisls / 5),
	i + 1,
	averageOcc); //fill with the mean over 5 lumisections, previous code was filling this histo only with last event of each 10th lumisection
	}
	}

[tvami]

What about blacklisting them in RelMon? Indeed I see they are in a blacklist:

cmssw/Utilities/RelMon/data/blacklist.txt

Lines 75 to 76 in 6d2f660

	Pixel/averageDigiOccupancy
	Pixel/AdditionalPixelErrors/byLumiErrors

averageDigiOccupancy is actually blacklisted even twice, the second one here

cmssw/Utilities/RelMon/data/blacklist.txt

Line 50 in 6d2f660

Pixel/averageDigiOccupancy

Maybe that could be changed to avgfedDigiOccvsLumi. Should I go ahead with this fix?

[mmusich]

Its typically straightforward to protect against generating nans.. what's the code that is doing this?

see #34900.

[jfernan2]

+1
Solved by #34900

[cmsbuild]

This issue is fully signed and ready to be closed.