Update edmStreamStallGrapher.py #15574
Conversation
The module was added to allow testing of stalling when running multiple threads.
The update deals with the framework change which changed when signals occur as well as allowing multiple modules to run on each stream.
|
A new Pull Request was created by @Dr15Jones (Chris Jones) for CMSSW_8_1_X. It involves the following packages: FWCore/Concurrency @cmsbuild, @smuzaffar, @Dr15Jones, @davidlange6 can you please review it and eventually sign? Thanks. cms-bot commands are list here #13028 |
|
please test |
|
+1 |
cmsbuild
added
core-approved
fully-signed
and removed
core-pending
pending-signatures
labels
|
The tests are being triggered in jenkins. |
|
This pull request is fully signed and it will be integrated in one of the next CMSSW_8_1_X IBs after it passes the integration tests. This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @smuzaffar |
| waitTime = time - streamTime[s] | ||
| if trans == kFinished: | ||
| if n != kSourceDelayedRead and n!=kSourceFindEvent and n!=kFinishInit: | ||
| del modulesOnStream[n] |
There was a problem hiding this comment.
should I be able to run this file on a tracer output file generated in CMSSW_8_1_X_2016-08-18-1100?
I'm getting an error
edmStreamStallGrapher.py", line 385, in <module>
stalledModules = findStalledModules(processingSteps, numStreams)
edmStreamStallGrapher.py", line 158, in findStalledModules
del modulesOnStream[n]
KeyError: 'siPixelDigis'
There was a problem hiding this comment.
It should work. Can you post the log file you used?
It is possible the MessageLogger changed order of some printouts.
There was a problem hiding this comment.
It's the same tracer.log file that I've sent you earlier by email, on cmsdev02
There was a problem hiding this comment.
I was wrong, this version can only properly parse files created since CMSSW_8_1_X_2016-08-22-1100.
There was a problem hiding this comment.
I was able to parse the log file after filtering out the following message
grep -v 'delayed processing event for module' just_tracer.log > noDelayed.log
There was a problem hiding this comment.
I found that your log file has one of the rare message inversions: there is a 'module finished' before the corresponding 'module starting' messages in the log output. I'll try putting in a protection for this.
There was a problem hiding this comment.
#15591 handles the message inversion cases
There was a problem hiding this comment.
You can see an updated image for that log at
https://www.dropbox.com/s/dyd1qgt1zh4h51z/stall.pdf?dl=0
I do notice that there are a large number of 'stalled' modules. Some of those are stream modules. My hypothesis is there are so many messages coming so fast that it can take an appreciable amount of time for some of the messages to be printed. This excessive timing is viewed as a module stall by the script.
It may be necessary to move to a dedicated Service to gather the information needed.
There was a problem hiding this comment.
looks nice.
Please send me the text file with stall times by module as well
|
+1 |
Updated edmStreamStallGrapher.py to be consistent with the recent changes to the framework. This includes