New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update edmStreamStallGrapher.py #15574
Update edmStreamStallGrapher.py #15574
Conversation
The module was added to allow testing of stalling when running multiple threads.
The update deals with the framework change which changed when signals occur as well as allowing multiple modules to run on each stream.
A new Pull Request was created by @Dr15Jones (Chris Jones) for CMSSW_8_1_X. It involves the following packages: FWCore/Concurrency @cmsbuild, @smuzaffar, @Dr15Jones, @davidlange6 can you please review it and eventually sign? Thanks. cms-bot commands are list here #13028 |
please test |
+1 |
The tests are being triggered in jenkins. |
This pull request is fully signed and it will be integrated in one of the next CMSSW_8_1_X IBs after it passes the integration tests. This pull request requires discussion in the ORP meeting before it's merged. @slava77, @davidlange6, @smuzaffar |
waitTime = time - streamTime[s] | ||
if trans == kFinished: | ||
if n != kSourceDelayedRead and n!=kSourceFindEvent and n!=kFinishInit: | ||
del modulesOnStream[n] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
should I be able to run this file on a tracer output file generated in CMSSW_8_1_X_2016-08-18-1100?
I'm getting an error
edmStreamStallGrapher.py", line 385, in <module>
stalledModules = findStalledModules(processingSteps, numStreams)
edmStreamStallGrapher.py", line 158, in findStalledModules
del modulesOnStream[n]
KeyError: 'siPixelDigis'
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should work. Can you post the log file you used?
It is possible the MessageLogger changed order of some printouts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's the same tracer.log file that I've sent you earlier by email, on cmsdev02
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was wrong, this version can only properly parse files created since CMSSW_8_1_X_2016-08-22-1100.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was able to parse the log file after filtering out the following message
grep -v 'delayed processing event for module' just_tracer.log > noDelayed.log
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I found that your log file has one of the rare message inversions: there is a 'module finished' before the corresponding 'module starting' messages in the log output. I'll try putting in a protection for this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#15591 handles the message inversion cases
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can see an updated image for that log at
https://www.dropbox.com/s/dyd1qgt1zh4h51z/stall.pdf?dl=0
I do notice that there are a large number of 'stalled' modules. Some of those are stream modules. My hypothesis is there are so many messages coming so fast that it can take an appreciable amount of time for some of the messages to be printed. This excessive timing is viewed as a module stall by the script.
It may be necessary to move to a dedicated Service to gather the information needed.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looks nice.
Please send me the text file with stall times by module as well
+1 |
Updated edmStreamStallGrapher.py to be consistent with the recent changes to the framework. This includes