[RFC] add commissioning_run as DQM runType and use it in SiStrip online DQM client
#34832
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34832/24613
|
|
A new Pull Request was created by @mmusich (Marco Musich) for master. It involves the following packages:
@andrius-k, @kmaeshima, @ErnestaP, @ahmad3213, @cmsbuild, @jfernan2, @rvenditti can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
please test |
|
-1 Failed Tests: RelVals-INPUT RelVals-INPUT
Expand to see more relval errors ...Comparison SummarySummary:
|
|
please test |
|
-1 Failed Tests: RelVals-INPUT RelVals-INPUT
Expand to see more relval errors ...Comparison SummarySummary:
|
|
@cmsbuild, please test |
|
+1 Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d1acaf/17718/summary.html Comparison SummarySummary:
|
|
Thanks @mmusich |
|
+1 |
cmsbuild
added
dqm-approved
fully-signed
and removed
dqm-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2) |
|
+1 |
|
OK @mmusich but do we have a run of type commissioning_run or we should create a fake one? |
That's the part I am ignorant about. From @rgerosa comments in the e-mail thread, it seems to me that the DAQ shifter should start a run which is then interpreted as |
This was referenced Aug 13, 2021
|
@mmusich sorry for the delay, I was in vacation in the past two weeks. I don't know the DQM details and how they can set/declare run-types. From the DAQ console (L0 FM) one can select different keys i.e. setups via which the DQM client can be configured. The idea is to have a ad-hoc key that does as you implemented i.e. HLT*, via which we can have events from random triggers being accepted by DQM. Therefore, I would guess that the DQM Run-Key selected by the DAQ should contain and propagate this info down the road. |
mmusich
added a commit
to mmusich/cmssw
that referenced
this pull request
Oct 15, 2021
mmusich
added a commit
to mmusich/cmssw
that referenced
this pull request
Oct 15, 2021
cmsbuild
added a commit
that referenced
this pull request
Oct 18, 2021
…o_to_commissioning_run provide a reconstruction scenario for the `commissioning_run` type introduced in #34832
marco-link
added a commit
to marco-link/cmssw
that referenced
this pull request
Oct 20, 2021
* Change to the new method writeOneIOV in PPSTimingCalibrationPCLHarvester * Fix 'U' mode is deprecated * Make the printouts a bit more concise * HLT migration to CMSSW_12_1_0_pre4 template * Replace unaligned pointer dereference with bitwise read operations * Remove unnecessary initialization * Update autoDQM.py @L1TMon needes by @hcal in @allForPrompt and it has been dropped from @common * fix issue with SiPixelBadModule_ may be used uninitialized in TSAN builds * Removing ECAL delete statements to revert to known working version. Added a single character in ECALTPG log output * Try to avoid the extended cell search * Code check * Better handling of strings in ExternalgeneratorFilter Now handle case where strings have embedded \n which must be properly propagated to the external workers. * Fix out of bounds error * Change to new method writeOneIOV * update hcal FED list for crate 3-FED readout * Add shim producers and example configs to allow refitting of muon tracks from AOD or MINIAOD using standard TrackRefitter * code formatting * code checks * modernize TrackExtraRekeyer * modernize TrackProducerFromPatMuons * modernize test muon track refitting configurations * add unit test for refitting from analysis data-tiers * Complete the extended check * Fix for o2o parameters from config database * Added test for ExternalGeneratorFilter and EvtGen together * correct Eid FED boolean assignment * Improve HGCAL geometry rendering in Fireworks * Remove double loop on detIds while selecting HGCAL components. This makes the HGCAL geometry rendering much much faster than before. * Use the newly introduced parameters/topology information to gather information about a specific HGCAL detId. Avoid detId unpacking. * Remove ad-hoc magic numbers while offsetting layers in the hadronic section. * Color coding for the Silicon sensor is now bound to their thickness, with the following color coding: * red is 120 um sensor * blue is 200 um sensor * green is 300 um sensor * Improve Trackster{Layer,Hits} Proxies * Position lines (centres of LayerClusters) [and seed lines for the `Hits` Proxy] are now registered to a single `TEveStraightLineSet`. They are, as a consequence, a single entity in the GUI, and the selection will highlight them all for each trackster, individually. * Edges are now partitioned in two `TEveStraightLineSet`: non-adjacent edges (spans more than one layer) and adjacent edges (between sibling layers). Non-adjacent edges are rendered in red, while adjacent edges in yellow. * Add transparency support for the `Layer` proxy. Still not fully automated, but working. * Keep color memory between consecutive switches between `heat map` and full color rendering. * Code format * Use assertEqual in testDelete() * Replace/delete a Path/EndPath/Task in Schedule too when replacing/deleting one in Process This should make the replacement behavior with respect to Schedule more consistent with the replacement of Sequences/Tasks/Modules in Paths/SequencesTasks. * Make printing of function parameters consistent * Fix bug the prevented anonymous namespace file path from being added * Add mote top level classes and functions * Include realistic CSCBadChambers in Run-3 MC GTs * Minor cleanup of the fishbone code * add stub of SiPixelLorentzAnglePCLWorker * add stub of SiPixelLorentzAnglePCLHarvester * fixed collection of tracks passed to track refitter * push code to write out payload, improve functionality, follow-up to review * clone instead of copy * update parseFwkJobReport to include SiPixelLA * modify writing logic post PR cms-sw#35556 * more clean-up * Add a protection for quasi-empty events Do not call the fishbone for quasi-empty events, with pixel hits only in the innermost layer. * add onlineBeamSpotESProducer to BeamSpot_cfi: fix general Online BS swap case * re-inroduce unit tests for visualization * Fix empty histograms in CSC L1T DQM * remove python2 contructs * Added unit test for edmPickEvents and dmCopyPickMerge * Update names of (two) parameters in mkFit README * Set beamspot in mkFit eventOfHits * Cleanup and code-format * Add a switch to L1THLTTauMatching to produce either PFTaus with a minimal contet or will fully filled datamembers Switch to CMSSW_11_3_X * Updates to deepTauID producer and PFTauSelector towards to enbling deepTauID at HLT * Address review comments * Keep just one customisation file for deepTauID at HLT * remove obsolete commands * Removed unused ObjectSelectorLegacy.h * Use mkfit::BeamSpot constructor * Fix for UBSAN crash As pointed out in: cms-sw#35605 (comment) * set equivalence of first/last bx labels used by different subsystems * Enable mkFit by default in 4+2 tracking iterations (initialStepPreSplitting, initialStep, highPtTripletStep, detachedQuadStep + detachedTripletStep, pixelLessStep) * formatting * Override beamspot parameter in customizeHLTIter0ToMkFit.py * Update SiPixelLorentzAnglePCLHarvester.cc * Update ALCARECOPromptCalibProdSiPixelLorentzAngle_cff.py * Update AlcaSiPixelLorentzAngleHarvester_cff.py * Remove unnecessary class members * Removed deprecated usages in PhysicsTools/CondLiteIO - made modules edm::one::EDAnalyzer - converted to use esConsumes * Use SimProducer+Analyzer to estimate material budget for HCAL * Code check * Remved unnecessary include * Modernize pythia6 related modules - moved away from legacy module - added consumes and esConsumes * Moved TauSpinnerFilter to be a stream module * Switch to @allForPrompt * adding safe for denominator=0 situation * Apply code cleanup Co-authored-by: Andrea Perrotta <perrotta@cern.ch> * Apply code format * Change default value of plotOffline flag * Change to the new method of using writeOneIOV * introduce unit tests for CondTools/SiPhase2Tracker * Added ConsumesCollector usage to ExternalDecayDriver Only Tauola needed it. * PPS bug fix of Run2 pixel topology * decouple writing legacy and HLT payloads on same sqlite file * Removed CMS deprecation warnings from MagneticField/Engine - converted legacy modules to one:: - added needed esConsumes calls * Remove CMS deprecation warnings in DetectorDescription/DDCMS - added esConsumes - changes simple modules to global:: * Added esConsumes to module in SimG4Core/DD4hepGeometry * Adding DNN variables for PFID for electrons in the dataformats * Evaluation of the PFID DNN also for photons * Added switch to disable the DNN PFID loading in Photon and Electron producers * Added switch to disable the DNN PFID loading in Photon and Electron producers Added expections for missing values in the DNN estimators Added switch to disable the DNN PFID loading in Photon and Electron producers Added expections for missing values in the DNN estimators * PF cluster isolations Added Pfiso variables for Photons Added PFiso electron variables for DNN * Added minmax scaler to DNN estimator * Moved PFcluster based isolation for electrons in GsfElectronAlgo * Added protection for HovEn and changed name of Photons variables Changed eta limits for photons also * Added default models and configurations for Run3 PFid Improved naming of parameters in PFEgammaFilters * Added initial values for PFEgammaFilters DNN thresholds * Added new scaler option for Photon PFID DNN Activate the PFID DNN by default * edm::FileInPath for model and scalers files * Disable PFiso computation if the DNN is disabled Code improvements * Added new DNN info in the PFCandidate dataformat * Improvements for TF sessions and fixes - Disabled missing PFCluster iso computation when DNN is not requested - Added Era modifier to activate the PFid DNN only in Run3_common era - Moved TF sessions in the DNNEstimator, so in the GlobalCache * Code improvements * Improved code for DNN estimator The duplicated code has been moved in EgammaTools/interface/EgammaDNNHelper.h. ElectronDNNEstimator and PhotonDNNestimator are now just used to create the correct list of variables and provide a function to select the model to be used for each candidate. * Removed logLevel from EgammaDNNHelper * Fixed bug in storing of DNN outputs in GsfElectronProducer * Added getters in PFCandidate, fixed missing scaling mode in EgammaDNNHelper * Disabled by default DNN Egamma PFID for pre4 * Enabled back Egm PFid by default; endRun -> endStream for sessions closing * Fixed code-checks, added const to PFCandidate getters * From atomic to unique_ptr for Tensorflow GraphDef * Intro unit test for PPSAssociationCuts * Made LeptonVertexSignificance independent of Event and EventSetup The constructor of the class would never be able to be called properly from cmsRun. * Add esConsumes calls to modules in PhysicsTools/PatUtils -Also switched to using std::unique_ptr * Use edm::ESGetToken when getting JME::JetResolution* - modules using the object were also moved away from inheriting from legacy modules * Fix CMS deprecation warnings in JetMETCorrections/Modules - moved modules away from inheriting from legacy modules - added esConsumes - switched to using edm::unique_ptr * Fix a crash caused by the missing luminosityBlockAuxiliary. A race can happen because local lock is unlocked when lsToStart is determined, and since this involves checking for marker files locally, another competing process can create EoL file since lock was released (and cause lsToStart to be increased above LS of the newly opened file). In case lsToStart is larger than ls, source would skip opening a lumisection before ending up processing events and this results in the lumi block related assertion. * Use correct record * load the onlineBeamSpotESProducer only if the modifier is called * Added new parameters from PPSAlignmentConfiguration to README. * Updated the worker and the harvester. - Replaced obsolete PPSAlignmentConfig and PPSAlignmentConfigESSource with PPSAlignmentConfiguration and PPSAlignmentConfigurationESSource, respectively. - Added support for new parameters in PPSAlignmentConfiguration, including: * min_RP_tracks_size, max_RP_tracks_size, * overwrite_sh_x, * new binning parameters. - Added merged results in the harvester, which can be saved to an SQLite file. - Huge refactor - Added fillDescriptions. - Shift ranges are now taken from the test config. - Added saving nice-looking cut plots. - Deleted redundant parameters for x relative alignment. * code-format * Updated the test folder. * Deleted unused cfi files. * Added RetrieveCTPPSRPAlignmentCorrectionsData. * p_x_diffFN_vs_x_N is now rebooked in the harvester. * Mode graph is now a TH1D. * Updated the AlignmentGlobal test (README and run_distributions_cfg.py). * Included PPS alignment in the PCL and added new matrix test. * Replaced writeOne with writeOneIOV in PPSAlignmentHarvester. * Clean up and avoid possible misuse of the numOfBins variable * Take Andrea's comments * Code format * Fixed typos * provide a reconstruction scenario for the commissioning_run type introduced in cms-sw#34832 * Improvements to LeptonVertexSignificance Changes suggested by @slava77 * Rearrange the code to enable future implementation of SimProducer class * adding ALCARECOTkAlCosmicsCosmicTF0T track collection in Millepede code * fix typo in _loadOnlineBeamSpotESProducer * removed unnecessary includes * Added esConsumes calls to Muon related code - Also updated some modules to no longer be legacy * switched to using std::unique_ptr for memory management * Avoid use of uninitalized values in GBRForestTools Calls to XMLElement::Query*Attribute do not set the passed in variable if there was a problem. So we either need to initialize the variable before hand or check for errors. * Flip the byte order * Apply code-format * updating hltDumpStream to work in python3 * Remove unnecessary type specifiers and clean * squashing migrate remaining DT calib code esConsumes migration Few corrections to make dtTTrigConstantShiftCorrection_cfg run fixes & more migration to esConsumes some more fix migrate test/DBTools to esConsumes Few additional fixes after testing code format review comments * Updated run3 hlt, express and prompt GTs in autoCond.py. * Made DQMHelper consistent with DQMEDAnalyzer - make sure typedefs will always be consistent - removed unnecessary copying of std::string * Use esConsumes for modules in Validation/EventGenerator * Use ESGetToken in getting objects from EventSetup * Code format * Take Amtti's suggestions * Adding SiPixelLorentzAngleRcd PCL tags * Adding CTPPS PCL tags * Code format * Converted the raw pointers that required delete to smart pointers. * Change config file to create a tag incl SiPixelLA and CTPPS * change bx constant to centrally defined value defined in DataFormats/uminosity/interface/LumiConstants.h * Add multiprocess support into compareHistograms Comparing full blown DQM files is painfully slow. This PR adds the possibility to run the comparisons over the histograms in parallel via a configurable parameter: `-j`, or the longer version `--num-processes`. The default behaviour is unchanged: using one process will run the current implementation. Selecting more than one process will trigger multiprocessing. The final sorting on the output lists is mandatory to keep a sane ordering within the saved ROOT files. It is useless for the DQMGUI, which does that already internally, but it's mandatory in the exploration of the base ROOT files. Usually the price to pay should be negligible compared to the speed-up gain thanks to the parallelism. Co-authored-by: cmsbuild <cmsbuild@cern.ch> Co-authored-by: Tamas <Tamas.Almos.Vami@cern.ch> Co-authored-by: Martin Grunewald <Martin.Grunewald@cern.ch> Co-authored-by: Christopher <krzysztof.misan@cern.ch> Co-authored-by: jfernan2 <Javier.Fernandez@cern.ch> Co-authored-by: mmusich <marco.musich@cern.ch> Co-authored-by: Abe Tishelman-Charny <abe2323@yahoo.com> Co-authored-by: Sunanda <sunanda.banerjee@cern.ch> Co-authored-by: Christopher Jones <chrisdjones15@gmail.com> Co-authored-by: Abhirami Harilal <abhi.harilal@gmail.com> Co-authored-by: Shahzad Malik Muzaffar <Shahzad.Malik.Muzaffar@cern.ch> Co-authored-by: Long <long.wang@cern.ch> Co-authored-by: Josh Bendavid <Josh.Bendavid@cern.ch> Co-authored-by: Giacomo Govi <giacomo.govi@cern.ch> Co-authored-by: Marco Rovere <marco.rovere@cern.ch> Co-authored-by: Matti Kortelainen <matti.kortelainen@cern.ch> Co-authored-by: Patrick Gartung <gartung@fnal.gov> Co-authored-by: Andrea Bocci <andrea.bocci@cern.ch> Co-authored-by: Sven Dildick <sven.dildick@cern.ch> Co-authored-by: David <David.Lange@cern.ch> Co-authored-by: Mario Masciovecchio <mario.masciovecchio@cern.ch> Co-authored-by: Liam <liamwezenbeek@hotmail.com> Co-authored-by: Michal Bluj <michal.bluj@cern.ch> Co-authored-by: Norraphat <srimanob@mail.cern.ch> Co-authored-by: Alejandro Sanchez <alejandro.sanchez@cern.ch> Co-authored-by: wweiphy <67766900+wweiphy@users.noreply.github.com> Co-authored-by: Christopher <52073715+ChrisMisan@users.noreply.github.com> Co-authored-by: Andrea Perrotta <perrotta@cern.ch> Co-authored-by: Fabrizio Ferro <fabferro@users.noreply.github.com> Co-authored-by: Davide Valsecchi <davide.valsecchi@cern.ch> Co-authored-by: Shilpi <shilpi.jain@cern.ch> Co-authored-by: Srecko Morovic <srecko.morovic@cern.ch> Co-authored-by: Mateusz <mateuszkocot99@gmail.com> Co-authored-by: Byeonghak Ko <quark2930@hotmail.com> Co-authored-by: Antonio Vagnerini <antonio.vagnerini@desy.de> Co-authored-by: Sam Harper <sam.j.harper@gmail.com> Co-authored-by: swmukher <s.mukherjee@cern.ch>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR description:
This PR adds a new DQM runType
commissioning_runin order to define a new clause in the SiStrip online client to be used when there are no physics triggers.PR validation:
at the moment none. to be further tested
if this PR is a backport please specify the original PR and why you need to backport that PR:
N/A