Adding FRD file header (DAQ) #27750
Adding FRD file header (DAQ) #27750
Conversation
identification and version fields, as well as metadata (file size and number of events), which are used for consistency checks in HLT nodes. In case of HLT (NFS) mode this is activated only when using file broker protocol v2. In this case accompanying JSON file is no longer present. In case of fileListMode of the FedRawDataSource and with FRDStramSource, it is autodetected based on reading the raw file. Commit includes a bugfix: use separate variable for reading second field in JSON file (affecting protocol with separate JSON index file; fortunately the bug has not been hit in the production system. Note: code-format-all has been executed and manually fixed due to changes being made prior to reformatting campaign
…adding after uint16 as reserved 16-bit field. Size is in either case 24, not 22. Use uint8 for consistency
in Streamer should also be migrated eventually.
- format update and change in initialization of the header structure
specifies [ls,run] tuple
…ition when inheriting. In case of 2 buffers (1 thread) only a single fd will be used with only sequential reading
- fix amount read into chunk in case of skipping header payload
|
@smorovic, CMSSW_11_0_X branch is closed for direct updates. cms-bot is going to move this PR to master branch. |
|
The code-checks are being triggered in jenkins. |
|
please test |
|
The tests are being triggered in jenkins. |
|
Comparison job queued. |
|
Comparison is ready Comparison Summary:
|
|
+1 |
|
+1 |
cmsbuild
added
core-approved
fully-signed
and removed
core-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2) |
|
+1 |
PR description:
Addition of file header to Fed Raw Data format. This header contains metadata description of the content (file size without header and number of events, LS and versioning). Aim is to replace separate JSON file which can take a lot of space in ramdisk with 2 MB huge pages (minimum per file).
File broker will switch protocol to v2 when detecting raw files with header, and Input source assumes i in this case . In addition there is also format autodetection when using fileListMode and in FRDSource, as well as generation support for the new format in the fake BU module.
Additional optimization is added: input source will, in case of using a single reader thread with double buffering (default mode), open raw file only once and reuse the same FD to read file sequentially until done (reducing number file operations performed).
PR validation:
This has been developed and tested in DAQ3VAL system.