Make ROOT file support concurrent LuminosityBlocks #22406
Conversation
IndexIntoFile was modified to handle the case where we are processing LuminosityBlocks concurrently and events from the next LuminosityBlock are being written before the endLuminosityBlock transition from the previous LuminosityBlock has been called. This was accomplished by adding a Lumi entry to runOrLumiEntries_ with an invalidEntry value which is there to keep track of the event entry range for the already written events for the previous LuminosityBlock.
RunLumiEventChecker can now be used in multi-threaded jobs (or to read files created by multi-threaded jobs) where the order of the transitions can change process to process. Also made RunLumiEventChecker a global module.
With the use of concurrent LuminosityBlocks, it is possible for an event of the next LuminosityBlock to be written to the ROOT file before the end LuminosityBlock transition has been seen by the OutputModule. This test checks that PoolOutputModule can properly write files in that case and that PoolSource can subsequently read them.
|
@wddgit please take a look |
|
The code-checks are being triggered in jenkins. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-22406/3646 |
|
A new Pull Request was created by @Dr15Jones (Chris Jones) for master. It involves the following packages: DataFormats/Provenance @cmsbuild, @smuzaffar, @Dr15Jones can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
@dan131riley does this modification also make it easier for you to implement the limited PoolOutputModule? |
|
please test |
|
The tests are being triggered in jenkins. |
|
+1 The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic: |
|
Comparison job queued. |
|
Comparison is ready Comparison Summary:
|
|
The differences with the base line are most likely caused by recent changes in the IB (since they also show up in other pull requests). Just to be safe, I'll rerun the tests |
|
please test |
|
The code-checks are being triggered in jenkins. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-22406/3824 |
|
Pull request #22406 was updated. @cmsbuild, @smuzaffar, @Dr15Jones can you please check and sign again. |
|
please test |
|
The tests are being triggered in jenkins. |
|
Comparison job queued. |
|
+1 |
cmsbuild
added
core-approved
fully-signed
and removed
core-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2) |
|
Comparison is ready Comparison Summary:
|
|
+1 |
The PoolOutputModule can now write files where an Event from the next LuminosityBlock is written to the output file before the end LuminosityBlock transition occurs. It can even handle interleaving Events from two different LuminosityBlocks before the previous LuminosityBlock ends.
The PoolSource can read files containing such interleaved Events. When reading back, it returns all the Events for one LuminosityBlock before going onto the next LuminosityBlock. [We already did such behavior in the cases where the output file was created by a job which read the LuminosityBlock from different files].