Update the `convertToRaw` script #39047

fwyzard · 2022-08-12T21:36:13Z

PR description:

Update the convertToRaw script:

change the default to 100 events per file, to match the value currently used online;
Improve the handling of the output and exit status of the subprocesses;
write a run####_cff.py file with the Source and Services to process the converted files;
add the -r/--range option to restrict the processing to a range of runs and lumisections;
add the -v/--verbose option to print additional message while preprocessing and converting the input files.

PR validation:

The new version runs and selects lumisections correctly.

Here is a sample output from running convertToRaw over one .root file:

$ convertToRaw -o /gpu_data/store/data/Run2022C/HLTPhysics/FED/v1 -r 34-65 root://eoscms.cern.ch//eos/cms/tier0/store/data/Run2022C/EphemeralHLTPhysics11/RAW/v1/000/357/329/00000/9e43f951-59ef-4dbf-85d1-28b5fa5f664f.root 
preprocessing input file root://eoscms.cern.ch//eos/cms/tier0/store/data/Run2022C/EphemeralHLTPhysics11/RAW/v1/000/357/329/00000/9e43f951-59ef-4dbf-85d1-28b5fa5f664f.root
  run 357329, lumisetion 29 is outside of the given range and will be skipped
  run 357329, lumisetion 30 is outside of the given range and will be skipped
  run 357329, lumisetion 31 is outside of the given range and will be skipped
  run 357329, lumisetion 32 is outside of the given range and will be skipped
  run 357329, lumisetion 33 is outside of the given range and will be skipped
  run 357329, lumisetion 34 with 463 events will be processed
  run 357329, lumisetion 35 with 431 events will be processed

found run 357329, lumi 34, with 463 events
cmsRun \
  /data/user/fwyzard/CMSSW_12_4_6/src/HLTrigger/Tools/python/convertToRaw.py \
  inputFiles=root://eoscms.cern.ch//eos/cms/tier0/store/data/Run2022C/EphemeralHLTPhysics11/RAW/v1/000/357/329/00000/9e43f951-59ef-4dbf-85d1-28b5fa5f664f.root \
  runNumber=357329 \
  lumiNumber=34 \
  eventsPerLumi=11655 \
  eventsPerFile=100 \
  outputPath=/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329_ls0034

found run 357329, lumi 35, with 431 events
cmsRun \
  /data/user/fwyzard/CMSSW_12_4_6/src/HLTrigger/Tools/python/convertToRaw.py \
  inputFiles=root://eoscms.cern.ch//eos/cms/tier0/store/data/Run2022C/EphemeralHLTPhysics11/RAW/v1/000/357/329/00000/9e43f951-59ef-4dbf-85d1-28b5fa5f664f.root \
  runNumber=357329 \
  lumiNumber=35 \
  eventsPerLumi=11655 \
  eventsPerFile=100 \
  outputPath=/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329_ls0035

And the corresponding /gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329_cff.py file:

import FWCore.ParameterSet.Config as cms

from EventFilter.Utilities.FedRawDataInputSource_cfi import source as _source
source = _source.clone(
    eventChunkSize = 200,   # MB
    eventChunkBlock = 200,  # MB
    numBuffers = 4,
    maxBufferedFiles = 4,
    fileListMode = True,
    fileNames = (
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0034_index000000.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0034_index000001.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0034_index000002.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0034_index000003.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0034_index000004.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0035_index000000.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0035_index000001.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0035_index000002.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0035_index000003.raw',
        '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1/run357329/run357329_ls0035_index000004.raw'
    )
)

from EventFilter.Utilities.EvFDaqDirector_cfi import EvFDaqDirector as _EvFDaqDirector
EvFDaqDirector = _EvFDaqDirector.clone(
    buBaseDir = '/gpu_data/store/data/Run2022C/HLTPhysics/FED/v1',
    runNumber = 357329
)

from EventFilter.Utilities.FastMonitoringService_cfi import FastMonitoringService as _FastMonitoringService
FastMonitoringService = _FastMonitoringService.clone()

If this PR will be backported please specify to which release cycle the backport is meant for:

To be backported to 12.4.x to convert data for use by the HLT.
See #39048.

Add the -r/--range option to restrict the processing to a range of runs and lumisections.

fwyzard · 2022-08-12T21:36:19Z

please test

fwyzard · 2022-08-12T21:38:28Z

please test

cmsbuild · 2022-08-12T21:43:42Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39047/31538

This PR adds an extra 16KB to repository

cmsbuild · 2022-08-12T21:44:08Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

HLTrigger/Tools (hlt)

@Martin-Grunewald, @missirol can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @silviodonato this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy, @rappoccio you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2022-08-13T05:03:44Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-689334/26793/summary.html
COMMIT: 57e087a
CMSSW: CMSSW_12_5_X_2022-08-12-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39047/26793/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3692476
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3692446
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

missirol · 2022-08-13T07:12:58Z

+hlt

cmsbuild · 2022-08-13T07:13:19Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

missirol · 2022-08-13T07:14:43Z

HLTrigger/Tools/scripts/convertToRaw

-events_per_file = 50
-events_per_lumi = 11650
+events_per_file = 100
+events_per_lumi = 11655


Just for my education: how was 11655 chosen?

That's how many events we can expect an HLT node to process in each lumisection under nominal conditions: 100 kHz / 200 nodes × 23.31 s = 11655 events

After processing each run, write a cff.py file with the Source and the necessary Services to process the converted files. Improve the handling of the output and exit status of the subprocesses, and add a -v/--verbose option to print additional message while preprocessing and converting the input files.

fwyzard · 2022-08-13T13:10:56Z

Fixed the case where the output has a single .raw file.

fwyzard · 2022-08-13T13:11:01Z

please test

cmsbuild · 2022-08-13T13:16:19Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39047/31547

This PR adds an extra 20KB to repository

cmsbuild · 2022-08-13T13:16:40Z

Pull request #39047 was updated. @Martin-Grunewald, @missirol can you please check and sign again.

cmsbuild · 2022-08-13T17:08:39Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-689334/26801/summary.html
COMMIT: 9a4b207
CMSSW: CMSSW_12_5_X_2022-08-13-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/39047/26801/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 51
DQMHistoTests: Total histograms compared: 3692476
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3692446
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 50 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 212 log files, 49 edm output root files, 51 DQM output files
TriggerResults: no differences found

missirol · 2022-08-13T18:18:29Z

HLTrigger/Tools/scripts/convertToRaw

-        f.close()
+        with open(eor_file, 'w') as file:
+            file.write('{\n   "data" : [ "%d", "%d", "%d", "%d" ],\n   "definition" : "%s",\n   "source" : "%s"\n}\n' % (summary['data'][0], summary['data'][1], summary['data'][2], summary['data'][3], summary['definition'], summary['source']))
+            file.close()


I thought with-open-as takes care of closing the file (so close wouldnt be needed).

it does, but I don't mind doing it explicitly

missirol · 2022-08-13T18:18:53Z

HLTrigger/Tools/scripts/convertToRaw

+from EventFilter.Utilities.FastMonitoringService_cfi import FastMonitoringService as _FastMonitoringService
+FastMonitoringService = _FastMonitoringService.clone()
+""" % ('\n'.join("        '" + f + "'," for f in converted_files), args.output_directory, run))
+        file.close()


I thought with-open-as takes care of closing the file (so close wouldnt be needed).

missirol · 2022-08-13T18:19:33Z

+hlt

Review comment is minor.

cmsbuild · 2022-08-13T18:20:00Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-08-14T01:03:15Z

+1

fwyzard added 2 commits August 12, 2022 23:33

Change the default to 100 events per file

40e96d5

Add an option to selct the run and lumi range

57e087a

Add the -r/--range option to restrict the processing to a range of runs and lumisections.

cmsbuild added this to the CMSSW_12_5_X milestone Aug 12, 2022

cmsbuild added code-checks-pending hlt-pending orp-pending pending-signatures tests-started labels Aug 12, 2022

fwyzard mentioned this pull request Aug 12, 2022

Update the convertToRaw script [12.4.x] #39048

Merged

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 12, 2022

cmsbuild added tests-approved and removed tests-started labels Aug 13, 2022

cmsbuild added fully-signed hlt-approved and removed hlt-pending pending-signatures labels Aug 13, 2022

missirol reviewed Aug 13, 2022

View reviewed changes

cmsbuild added code-checks-pending and removed hlt-approved fully-signed tests-approved code-checks-approved labels Aug 13, 2022

fwyzard force-pushed the update_convertToRaw_125x branch from 8317e73 to 9a4b207 Compare August 13, 2022 13:10

cmsbuild added code-checks-pending tests-pending and removed tests-started code-checks-approved labels Aug 13, 2022

cmsbuild added tests-started and removed tests-pending labels Aug 13, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Aug 13, 2022

cmsbuild added tests-approved and removed tests-started labels Aug 13, 2022

missirol reviewed Aug 13, 2022

View reviewed changes

cmsbuild added fully-signed hlt-approved and removed hlt-pending pending-signatures labels Aug 13, 2022

cmsbuild added orp-approved and removed orp-pending labels Aug 14, 2022

cmsbuild merged commit f44cb5d into cms-sw:master Aug 14, 2022

cmsbuild mentioned this pull request Aug 14, 2022

fixup reading histograms in TrackTimingTable in trackingPlots #39054

Merged

fwyzard deleted the update_convertToRaw_125x branch August 16, 2022 05:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update the `convertToRaw` script #39047

Update the `convertToRaw` script #39047

fwyzard commented Aug 12, 2022 •

edited

fwyzard commented Aug 12, 2022

fwyzard commented Aug 12, 2022

cmsbuild commented Aug 12, 2022

cmsbuild commented Aug 12, 2022

cmsbuild commented Aug 13, 2022

missirol commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

missirol Aug 13, 2022

fwyzard Aug 13, 2022

fwyzard commented Aug 13, 2022

fwyzard commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

missirol Aug 13, 2022

fwyzard Aug 13, 2022

missirol Aug 13, 2022

missirol commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

qliphy commented Aug 14, 2022

Update the convertToRaw script #39047

Update the convertToRaw script #39047

Conversation

fwyzard commented Aug 12, 2022 • edited

PR description:

PR validation:

If this PR will be backported please specify to which release cycle the backport is meant for:

fwyzard commented Aug 12, 2022

fwyzard commented Aug 12, 2022

cmsbuild commented Aug 12, 2022

cmsbuild commented Aug 12, 2022

cmsbuild commented Aug 13, 2022

Comparison Summary

missirol commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

missirol Aug 13, 2022

Choose a reason for hiding this comment

fwyzard Aug 13, 2022

Choose a reason for hiding this comment

fwyzard commented Aug 13, 2022

fwyzard commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

Comparison Summary

missirol Aug 13, 2022

Choose a reason for hiding this comment

fwyzard Aug 13, 2022

Choose a reason for hiding this comment

missirol Aug 13, 2022

Choose a reason for hiding this comment

missirol commented Aug 13, 2022

cmsbuild commented Aug 13, 2022

qliphy commented Aug 14, 2022

Update the `convertToRaw` script #39047

Update the `convertToRaw` script #39047

fwyzard commented Aug 12, 2022 •

edited