metadata JSON files produced for empty lumisections in DAQ (75X) #10807

smorovic · 2015-08-16T17:13:56Z

Implementation of JSON metadata output for each lumisection in Filter Units and several other smaller changes and bugfixes:

file distribution scheme in ramdisk previously allowed skipping lumisections with no data. This is now modified so that each process must transition through each lumisection starting from one first seen in lock file in BU ramdisk.
correponding changes in DAQ (streamer) output module, and DQMHIstogram, L1/HLT rates stream producers to output metadata json for empty lumisections
avoid casting -1 to uint32 in Trigger JSON module, which converts it in the valid range of the checksum (-1 is intended as true negative in JSON to signal that checksumming is disabled in particular cases)
added "maxBufferedFiles" parameter which limits the number of currently open files (useful in case of more than two input buffers used)
exception thrown when "_" (underscore) is used as stream name because this breaks file naming convention in DAQ (and breaks output merging by hltd)
updated fake BU code with latest json format used by DAQ BU application
removed old unused test mode code in input source and DaqDirector
metadata JSON files produced for empty lumisections in DAQ (74X) #10796 is 74X equivalent of this PR

(modifiable with maxOpenFiles parameter).

… empty lumisection mode *checking empty LS mode parameter in DQM and Streamer (Reco) outputs

*case with bumping LS but no new file needs readIndex 0

…der BU application)

fix: !shoudWriteFiles --> shouldWriteFiles fix: name of jsndata file for L1 stream

…ion, start from first one discovered by the process

…d is within a valid range of the checksum . IntJ is also internally 32-bit int, so instead use unsigned int directly and by default use 1

…le python label) as it is reserved by DAQ filename convention

cmsbuild · 2015-08-16T17:16:12Z

A new Pull Request was created by @smorovic (Srecko Morovic) for CMSSW_7_5_X.

metadata JSON files produced for empty lumisections in DAQ (75X)

It involves the following packages:

DQMServices/Components
EventFilter/Utilities
HLTrigger/JSONMonitoring

@perrotta, @cmsbuild, @cvuosalo, @fwyzard, @emeschi, @Martin-Grunewald, @deguio, @slava77, @mommsen, @danduggan can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @barvic this is something you requested to watch as well.
You can sign-off by replying to this message having '+1' in the first line of your reply.
You can reject by replying to this message having '-1' in the first line of your reply.
If you are a L2 or a release manager you can ask for tests by saying 'please test' in the first line of a comment.
@Degano you are the release manager for this.
You can merge this pull request by typing 'merge' in the first line of your comment.

Martin-Grunewald · 2015-08-16T18:50:45Z

please test

cmsbuild · 2015-08-16T18:51:20Z

The tests are being triggered in jenkins.

cmsbuild · 2015-08-17T15:44:06Z

-1
Tested at: d208fd0
When I ran the RelVals I found an error in the following worklfows:
50202.0 step2

runTheMatrix-results/50202.0_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50/step2_TTbar_13+TTbar_13+DIGIUP15_PU50+RECOUP15_PU50+HARVESTUP15_PU50.log
----- Begin Fatal Exception 17-Aug-2015 17:25:24 CEST-----------------------
An exception of category 'Configuration' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

4.22 step1

DAS Error

140.53 step1

DAS Error

1000.0 step1

DAS Error

1001.0 step1

DAS Error

1003.0 step1

DAS Error

you can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10807/7385/summary.html

Martin-Grunewald · 2015-08-18T04:42:53Z

please test

cmsbuild · 2015-08-18T04:46:54Z

The tests are being triggered in jenkins.

cmsbuild · 2015-08-18T06:09:13Z

+1
Tested at: d208fd0
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10807/7406/summary.html

cmsbuild · 2015-08-18T06:55:17Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10807/7406/summary.html

danduggan · 2015-08-18T07:00:45Z

+1

On Tue, Aug 18, 2015 at 8:55 AM, cmsbuild notifications@github.com wrote:

Comparison is ready

https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-10807/7406/summary.html

—
Reply to this email directly or view it on GitHub
#10807 (comment).

Martin-Grunewald · 2015-08-18T07:04:18Z

+1

cmsbuild · 2015-08-18T07:06:50Z

This pull request is fully signed and it will be integrated in one of the next CMSSW_7_5_X IBs once checked with relvals in the development release cycle of CMSSW (tests are also fine). This pull request requires discussion in the ORP meeting before it's merged. @davidlange6, @Degano, @smuzaffar

metadata JSON files produced for empty lumisections in DAQ (75X)

smorovic added 17 commits August 14, 2015 23:19

*produce metadata json files for empty lumisections

e6fc5ff

*always produce PB file even if lumisection is empty

3bc9e30

start of event processing signal sent to hltd

d6b1ce3

output should still be suppressed if there was exception or a crash

16ac0f4

*N-buffering allowed to keep open max 2 files at the time by default

7ea79b1

(modifiable with maxOpenFiles parameter).

maxBufferedFiles

854a2e9

write processing file flag only initially

a625706

DQM (ROOT/PB) files not produced in empty lumisections

4ec8730

only json file output when 0 processed events, parameter to configure…

6072468

… empty lumisection mode *checking empty LS mode parameter in DQM and Streamer (Reco) outputs

EvFOutputModule gets same treatment as legacy name in ustate monitoring

f9228f8

create processing notification only in emptyLS mode

4278bf2

*removed test mode, and jump* variables

399653d

*case with bumping LS but no new file needs readIndex 0

byte size in EoLS and index jsn in fakeBU (as added in the event buil…

bc11e5b

…der BU application)

compilation fixes, removing test mode code which no longer works

437abac

fix: !shoudWriteFiles --> shouldWriteFiles fix: name of jsndata file for L1 stream

*changes to input lumisection discovery. cycle through every lumisect…

f33516d

…ion, start from first one discovered by the process

1 is default value of adler32, while 32-bit -1 interpreted as unsigne…

ced5401

…d is within a valid range of the checksum . IntJ is also internally 32-bit int, so instead use unsigned int directly and by default use 1

throw exception if underscore is used in the stream name (output modu…

0476b97

…le python label) as it is reserved by DAQ filename convention

cmsbuild added this to the Next CMSSW_7_5_X milestone Aug 16, 2015

cmsbuild added reconstruction-pending dqm-pending hlt-pending daq-pending pending-signatures tests-pending orp-pending comparison-pending labels Aug 16, 2015

smorovic mentioned this pull request Aug 16, 2015

metadata JSON files produced for empty lumisections in DAQ (76X) #10808

Merged

cmsbuild added daq-approved tests-started and removed tests-pending labels Aug 17, 2015

cmsbuild added tests-rejected and removed tests-started labels Aug 17, 2015

cmsbuild added tests-pending tests-started and removed tests-rejected tests-pending labels Aug 18, 2015

cmsbuild added tests-approved and removed tests-started labels Aug 18, 2015

cmsbuild added comparison-available and removed comparison-pending labels Aug 18, 2015

cmsbuild added dqm-approved and removed dqm-pending labels Aug 18, 2015

cmsbuild added hlt-approved fully-signed and removed hlt-pending pending-signatures labels Aug 18, 2015

davidlange6 added a commit that referenced this pull request Aug 19, 2015

Merge pull request #10807 from smorovic/empty-LS-mode-75X

634b4c4

metadata JSON files produced for empty lumisections in DAQ (75X)

davidlange6 merged commit 634b4c4 into cms-sw:CMSSW_7_5_X Aug 19, 2015

smorovic deleted the empty-LS-mode-75X branch November 13, 2015 10:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

metadata JSON files produced for empty lumisections in DAQ (75X) #10807

metadata JSON files produced for empty lumisections in DAQ (75X) #10807

smorovic commented Aug 16, 2015

cmsbuild commented Aug 16, 2015

Martin-Grunewald commented Aug 16, 2015

cmsbuild commented Aug 16, 2015

cmsbuild commented Aug 17, 2015

Martin-Grunewald commented Aug 18, 2015

cmsbuild commented Aug 18, 2015

cmsbuild commented Aug 18, 2015

cmsbuild commented Aug 18, 2015

danduggan commented Aug 18, 2015

Martin-Grunewald commented Aug 18, 2015

cmsbuild commented Aug 18, 2015

metadata JSON files produced for empty lumisections in DAQ (75X) #10807

metadata JSON files produced for empty lumisections in DAQ (75X) #10807

Conversation

smorovic commented Aug 16, 2015

cmsbuild commented Aug 16, 2015

Martin-Grunewald commented Aug 16, 2015

cmsbuild commented Aug 16, 2015

cmsbuild commented Aug 17, 2015

Martin-Grunewald commented Aug 18, 2015

cmsbuild commented Aug 18, 2015

cmsbuild commented Aug 18, 2015

cmsbuild commented Aug 18, 2015

danduggan commented Aug 18, 2015

Martin-Grunewald commented Aug 18, 2015

cmsbuild commented Aug 18, 2015