TestDQMOnlineClient-beamhlt_dqm_sourceclient silently fails in IBs

[mmusich]

Casually looking at the logs of the DQM/Integration unit tests, I noticed that the test TestDQMOnlineClient-beamhlt_dqm_sourceclient fails silently:

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/el8_amd64_gcc12/CMSSW_13_3_X_2023-10-24-2300/unitTestLogs/DQM/Integration#/2700-2700

the exception message is:

25-Oct-2023 03:52:39 CEST  Can't deserialize event or registry data: An exception of category 'FatalRootError' occurred.
   Additional Info:
      [a] Fatal Root Error: @SUB=TBufferFile::ReadClassBuffer
Could not find the StreamerInfo for version 3 of the class edm::Wrapper<FEDRawDataCollection>, object skipped at offset 386

but then:

---> test TestDQMOnlineClient-beamhlt_dqm_sourceclient succeeded

The error message looks reminiscent of #41348, but perhaps core experts can chime in here.
Certainly it's worrisome that the test fails without being registered.
Additionally I think the streamer files used in input are outdated (see also cms-data/DQM-Integration#3).
Cc: @francescobrivio

[cmsbuild]

A new Issue was created by @mmusich Marco Musich.

@makortel, @Dr15Jones, @sextonkennedy, @smuzaffar, @rappoccio, @antoniovilela can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[francescobrivio]

Thanks Marco for opening the issue!

Additionally I think the streamer files used in input are outdated (see also cms-data/DQM-Integration#3).
Cc: @francescobrivio

Regarding this: I'm about to opens PRs to update the streamer files (cms-data) and the unitTest (cmssw) which, btw works smoothly with the new streamer files.

[mmusich]

I'm about to opens PRs to update the streamer files (cms-data) and the unitTest (cmssw) which, btw works smoothly with the new streamer files.

I think we should first strive to understand why the failure is not caught before sweeping it under the carpet.

[francescobrivio]

I'm about to opens PRs to update the streamer files (cms-data) and the unitTest (cmssw) which, btw works smoothly with the new streamer files.

I think we should first strive to understand why the failure is not caught before sweeping it under the carpet.

Sure I agree!
My idea was to have the unitTest run twice for the moment:

• with new streamer files
• with old streamer files --> so we can keep debugging this issue

[francescobrivio]

In #43110 this exactly this:

• updated the TestDQMOnlineClient-beamhlt_dqm_sourceclient unitTest to use the new streamer files
• added TestDQMOnlineClient-beamhlt_dqm_sourceclient-legacy which is exactly identical to the old one, i.e. it shows the FatalRootError exception, but the unitTest is deemed successful by the bot

[mmusich]

actually looks like DQMStreamerReader is expected to catch the exception and do nothing about it:

cmssw/DQMServices/StreamerIO/plugins/DQMStreamerReader.cc

Lines 149 to 160 in 7e21693

	if (std::filesystem::exists(p)) {
	try {
	openFileImp_(currentLumi);
	return true;
	} catch (const cms::Exception& e) {
	fiterator_.logFileAction(std::string("Can't deserialize registry data (in open file): ") + e.what(), p);
	fiterator_.logLumiState(currentLumi, "error: data file corrupted");
	closeFileImp_("data file corrupted");
	return false;
	}
	} else {

[makortel]

assign dqm

[cmsbuild]

New categories assigned: dqm

@rvenditti,@syuvivida,@tjavaid,@nothingface0,@antoniovagnerini you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

The error message looks reminiscent of #41348, but perhaps core experts can chime in here.

Streamer files do not support schema evolution.

[mmusich]

Streamer files do not support schema evolution.

good point... which makes the unit test based on streamer files rather brittle...

[francescobrivio]

So the only way out of this is to keep updating (frequently) the streamer files used in the unitTests?

[mmusich]

So the only way out of this is to keep updating (frequently) the streamer files used in the unitTests?

I don't know how frequently the backward compatibility is broken, but I guess few times per year? In any case this is aggravated by the fact that we cannot catch it by looking at the unit test results in IB logs, since as pointed out it just silently fails.

[smuzaffar]

Can we add a configuration flag to fail instead of silently ignoring the exception? The flag should be disabled by default and only unit test can enable it

[mmusich]

Can we add a configuration flag to fail instead of silently ignoring the exception? The flag should be disabled by default and only unit test can enable it

that's fine by me, if @cms-sw/dqm-l2 agree.

[mmusich]

this is fixed at #45231 + cms-data/DQM-Integration#8

[cmsbuild]

cms-bot internal usage