CMSSW Config Tracing Tool #25762
CMSSW Config Tracing Tool #25762
Conversation
Using the python trace module, which is rather slow. Also ignoring stdlib/FWCore seems to not work atm.
There is not much performance to be gained by not doing the call tracing. The calltree output now be processed with the FlameGraph tool.
This is related to the PSetHistory mode in CMSSW, but tracks something slightly different: The goal is to keep track of all places in the configuration that actually affect the process running in the end. A the moment there is no output, but the _trace_events on each Sequence are populated with the lines of code that modified this Sequence. Since all the constituents are preserved in the _seq itself, this should be enough to reconstruct the entire configuration tree and assign a source location to each item.
The output is for each object remaining in the process the place where this object is used (can be multiple). This is not very intuitive, but should be visualizable as an annotation on the source code, as 'why is this object needed?'
Still not able to see through 'convertToUnscheduled', we' have to knock that one out.
Also there is still sometimes issues with database locking.
|
The code-checks are being triggered in jenkins. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25762/8176 |
|
The code-checks are being triggered in jenkins. |
(though not tested on python3)
|
The code-checks are being triggered in jenkins. |
|
@fabiocos I changed the python lines. |
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-25762/8363 |
|
Pull request #25762 was updated. @cmsbuild, @kmaeshima, @andrius-k, @jfernan2, @schneiml can you please check and sign again. |
|
please test |
|
The tests are being triggered in jenkins. |
|
Comparison job queued. |
|
Comparison is ready Comparison Summary:
|
|
+1 |
cmsbuild
added
dqm-approved
fully-signed
and removed
dqm-pending
pending-signatures
labels
|
This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2) |
|
+1 |
This PR adds two tools that help with understanding CMSSW configurations.
cmsswFiletrace.pyuses the Pythonsettracefeature to track call graphs and therefore imports etc.cmsswConfigtrace.pyuses hooks to attach additional information to the config objects, like theEnablePSetHistorything used to do.cmsswConfigtraceis more advanced and comes with a small web frontend to read the saved information.I put the tools into
DQMOffline/Configurationsince I intend to use them for DQM offline configuration, though they are rather generic.