MessageLogger parser script #29758

fabiocos · 2020-05-07T14:47:36Z

PR description:

This PR contains a refreshed version of an old python utility of mine that I have used for years to extract in a selective way MessageLogger information from log files. As I am recently sharing it with others to manipulate geometry validation dumps, I propose here in case it might be considered of more general interest. FWCore/MessageLogger/scripts looks a natural location to me, but of course it might be moved elsewhere in case.

PR validation:

All options and combinations I have tested look to work as expected.

cmsbuild · 2020-05-07T14:48:06Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-05-07T14:54:24Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29758/15204

This PR adds an extra 12KB to repository

cmsbuild · 2020-05-07T14:54:45Z

A new Pull Request was created by @fabiocos (Fabio Cossutti) for master.

It involves the following packages:

FWCore/MessageLogger

@makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please review it and eventually sign? Thanks.
@makortel, @wddgit this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

fabiocos · 2020-05-07T14:56:52Z

please test

cmsbuild · 2020-05-07T14:57:12Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6159/console Started: 2020/05/07 17:06

makortel · 2020-05-07T15:14:50Z

@fabiocos I suppose you have tested the script with python3? Could you give a short example of what it does?

cmsbuild · 2020-05-07T16:13:52Z

+1
Tested at: 8c5555b
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eb5603/6159/summary.html
CMSSW: CMSSW_11_1_X_2020-05-07-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-05-07T16:13:54Z

Comparison job queued.

fabiocos · 2020-05-07T16:39:13Z

@makortel yes, it compiles with python3, and in CMSSW_11_1_PY3_X_2020-05-06-2300 it behaves as expected. Just to make a simple example:

%MSG-i DD4hep_TestMTDPath:  DD4hep_TestMTDIdealGeometry:testETL  07-May-2020 14:02:48 CEST Run: 1 Event: 1
 - OCMS[0]/CMSE[1]/CALO[1]/CALOEC[2]/CALOECTSFront[1]/CALOECFront[1]/EndcapTimingLayer[1]/Disc1[1]/Ring11Disc1[1]/EModule11Disc1[164]/EModule11Disc1Timingwafer[1]/
%MSG
%MSG-i DD4hep_TestMTDPath:  DD4hep_TestMTDIdealGeometry:testETL  07-May-2020 14:02:48 CEST Run: 1 Event: 1
 - OCMS[0]/CMSE[1]/CALO[1]/CALOEC[2]/CALOECTSFront[1]/CALOECFront[1]/EndcapTimingLayer[1]/Disc1[1]/Ring11Disc1[1]/EModule11Disc1[164]/EModule11Disc1Timingwafer[1]/EModule11Disc1T
imingactive[1]/
%MSG
%MSG-i DD4hep_TestMTDNumbering:  DD4hep_TestMTDIdealGeometry:testETL  07-May-2020 14:02:48 CEST Run: 1 Event: 1
[MTDDetId::print]  0110 0011 0000 1011 0101 0010 0000 0000
 rawId       : 0x630b5200 / 1661686272
 bits[0:24]  : 010b5200
 Detector        : 6
 SubDetector     : 1
 MTD subdetector : 2
 ETL 
 Side        : 0
 Ring        : 11     Disc/Side/Sector = 2 0 3
 Module      : 164
 Module type : 0

%MSG
%MSG-i DD4hep_TestMTDPosition:  DD4hep_TestMTDIdealGeometry:testETL  07-May-2020 14:02:48 CEST Run: 1 Event: 1
Solid shape name: Box
Box dimensions:      24.500000      47.500000       0.150000
Translation vector components:   -1162.867432    -220.026329   -3012.150000 
Rotation matrix components:       0.185912      -0.982566       0.000000       0.982566       0.185912       0.000000       0.000000      -0.000000       1.000000
Center global   =   -1162.867432   -220.026329  -3012.150000 r =    1183.500000 phi =    -169.285717
Corner 1 local  =      24.500000     47.500000      0.150000 DeltaR =      53.446445
Corner 1 global =   -1111.640691   -235.268409  -3012.000000 DeltaR =      53.446445


%MSG
%MSG-i DD4hep_TestMTDPath:  DD4hep_TestMTDIdealGeometry:testETL  07-May-2020 14:02:48 CEST Run: 1 Event: 1
 - OCMS[0]/CMSE[1]/CALO[1]/CALOEC[2]/CALOECTSFront[1]/CALOECFront[1]/EndcapTimingLayer[1]/Disc1[1]/Ring11Disc1[1]/EModule11Disc1[164]/EModule11Disc1FSide[1]/
%MSG

can be manipulated to extract with

edmMLParser -c DD4hep_TestMTDPath,DD4hep_TestMTDPosition cmsExec_testMTDinDD4hep_20200507_140210.log

an output like


 - OCMS[0]/CMSE[1]/Tracker[1]/BarrelTimingLayer[1]/Layer1[1]/Rod1[1]/BModule1Layer1PositiveZ[1]/BModule1Layer1PositiveZTimingwafer[1]/BModule1Layer1Timingactive[2]/
Solid shape name: Box
Box dimensions:      28.800000       1.575000       1.875000
Translation vector components:    1166.020836     140.874621       1.631200 
Rotation matrix components:      -0.119944       0.992781      -0.000000      -0.000000      -0.000000       1.000000       0.992781       0.119944      -0.000000
Center global   =    1166.020836    140.874621      1.631200 r =    1174.500000 phi =       6.888890
Corner 1 local  =      28.800000      1.575000      1.875000 DeltaR =      28.903914
Corner 1 global =    1164.427903    169.691599      3.206200 DeltaR =      28.903914

You may select specified categories, level of verbosity, optionally print the message label, or the Begin event statements in case you want to correlate the messages with events. It's nothing earthshaking, but it may be useful for very large outputs where to search for limited instances, or to combine a subset of categories to build customised reports.

fabiocos · 2020-05-07T16:39:48Z

18:37 cmsdev25 1501> edmMLParser -h
usage: edmMLParser [-h] [-c CATEGORY_NAME] [-b] [-m] [-v VERBOSITYLEVEL] file

edmMLParser <options> <input file>

positional arguments:
  file                  input file name

optional arguments:
  -h, --help            show this help message and exit
  -c CATEGORY_NAME, --category CATEGORY_NAME
                        List of MessageLogger categories to be searched, separated by ","
  -b, --print-begin-event
                        Print the begin event record
  -m, --print-message-label
                        Print the MessageLogger label for the selected message
  -v VERBOSITYLEVEL, --select-verbosity-level VERBOSITYLEVEL
                        Select a particular verbosity level for the chosen category, possible values are: !iwe

cmsbuild · 2020-05-07T17:40:52Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eb5603/6159/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2697527
DQMHistoTests: Total failures: 31
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2697177
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

makortel

Thanks for the example, looks useful indeed.

FWCore/MessageLogger/scripts/edmMLParser

cmsbuild · 2020-05-08T09:06:01Z

The code-checks are being triggered in jenkins.

fabiocos · 2020-05-08T09:06:52Z

@makortel thank you for the suggestion, I have updated the code

cmsbuild · 2020-05-08T09:11:48Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-29758/15223

This PR adds an extra 12KB to repository

cmsbuild · 2020-05-08T09:12:11Z

Pull request #29758 was updated. @makortel, @smuzaffar, @cmsbuild, @Dr15Jones can you please check and sign again.

fabiocos · 2020-05-08T09:13:58Z

please test

cmsbuild · 2020-05-08T09:14:19Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6172/console Started: 2020/05/08 11:14

cmsbuild · 2020-05-08T10:28:09Z

+1
Tested at: 58aa923
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eb5603/6172/summary.html
CMSSW: CMSSW_11_1_X_2020-05-07-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-05-08T10:28:11Z

Comparison job queued.

cmsbuild · 2020-05-08T11:59:40Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eb5603/6172/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2697527
DQMHistoTests: Total failures: 20
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2697188
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

makortel · 2020-05-08T16:56:54Z

+1

Thanks

cmsbuild · 2020-05-08T16:57:20Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-05-08T19:53:20Z

+1

Add MessageLogger parser to edm scripts

8c5555b

cmsbuild added this to the CMSSW_11_1_X milestone May 7, 2020

cmsbuild added code-checks-pending comparison-pending core-pending orp-pending pending-signatures tests-pending labels May 7, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels May 7, 2020

cmsbuild added tests-started and removed tests-pending labels May 7, 2020

cmsbuild added tests-approved and removed tests-started labels May 7, 2020

cmsbuild added comparison-available and removed comparison-pending labels May 7, 2020

makortel reviewed May 8, 2020

View reviewed changes

FWCore/MessageLogger/scripts/edmMLParser Outdated Show resolved Hide resolved

FWCore/MessageLogger/scripts/edmMLParser Outdated Show resolved Hide resolved

FWCore/MessageLogger/scripts/edmMLParser Outdated Show resolved Hide resolved

Address comments by M. Kortelainen

58aa923

cmsbuild removed the code-checks-approved label May 8, 2020

cmsbuild added comparison-pending tests-pending labels May 8, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels May 8, 2020

cmsbuild added tests-started and removed tests-pending labels May 8, 2020

cmsbuild added tests-approved and removed tests-started labels May 8, 2020

cmsbuild added comparison-available and removed comparison-pending labels May 8, 2020

cmsbuild added core-approved fully-signed and removed core-pending pending-signatures labels May 8, 2020

cmsbuild added orp-approved and removed orp-pending labels May 8, 2020

cmsbuild merged commit 131e0c0 into cms-sw:master May 8, 2020

fabiocos deleted the fc-mlparser branch May 11, 2020 08:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MessageLogger parser script #29758

MessageLogger parser script #29758

fabiocos commented May 7, 2020 •

edited

cmsbuild commented May 7, 2020

cmsbuild commented May 7, 2020

cmsbuild commented May 7, 2020

fabiocos commented May 7, 2020

cmsbuild commented May 7, 2020 •

edited

makortel commented May 7, 2020

cmsbuild commented May 7, 2020

cmsbuild commented May 7, 2020

fabiocos commented May 7, 2020

fabiocos commented May 7, 2020

cmsbuild commented May 7, 2020

makortel left a comment

cmsbuild commented May 8, 2020

fabiocos commented May 8, 2020

cmsbuild commented May 8, 2020

cmsbuild commented May 8, 2020

fabiocos commented May 8, 2020

cmsbuild commented May 8, 2020 •

edited

cmsbuild commented May 8, 2020

cmsbuild commented May 8, 2020

cmsbuild commented May 8, 2020

makortel commented May 8, 2020

cmsbuild commented May 8, 2020

silviodonato commented May 8, 2020

MessageLogger parser script #29758

MessageLogger parser script #29758

Conversation

fabiocos commented May 7, 2020 • edited

PR description:

PR validation:

cmsbuild commented May 7, 2020

cmsbuild commented May 7, 2020

cmsbuild commented May 7, 2020

fabiocos commented May 7, 2020

cmsbuild commented May 7, 2020 • edited

makortel commented May 7, 2020

cmsbuild commented May 7, 2020

cmsbuild commented May 7, 2020

fabiocos commented May 7, 2020

fabiocos commented May 7, 2020

cmsbuild commented May 7, 2020

makortel left a comment

Choose a reason for hiding this comment

cmsbuild commented May 8, 2020

fabiocos commented May 8, 2020

cmsbuild commented May 8, 2020

cmsbuild commented May 8, 2020

fabiocos commented May 8, 2020

cmsbuild commented May 8, 2020 • edited

cmsbuild commented May 8, 2020

cmsbuild commented May 8, 2020

cmsbuild commented May 8, 2020

makortel commented May 8, 2020

cmsbuild commented May 8, 2020

silviodonato commented May 8, 2020

fabiocos commented May 7, 2020 •

edited

cmsbuild commented May 7, 2020 •

edited

cmsbuild commented May 8, 2020 •

edited