Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 345 lines (270 sloc) 10.79 kB
a85bd0c Initial commit
Jeff Muizelaar authored
1 ## statprof.py
9ae3cc6 @bos Track line numbers properly.
authored
2 ## Copyright (C) 2012 Bryan O'Sullivan <bos@serpentine.com>
a85bd0c Initial commit
Jeff Muizelaar authored
3 ## Copyright (C) 2011 Alex Fraser <alex at phatcore dot com>
4 ## Copyright (C) 2004,2005 Andy Wingo <wingo at pobox dot com>
5 ## Copyright (C) 2001 Rob Browning <rlb at defaultvalue dot org>
6
7 ## This library is free software; you can redistribute it and/or
8 ## modify it under the terms of the GNU Lesser General Public
9 ## License as published by the Free Software Foundation; either
10 ## version 2.1 of the License, or (at your option) any later version.
11 ##
12 ## This library is distributed in the hope that it will be useful,
13 ## but WITHOUT ANY WARRANTY; without even the implied warranty of
14 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
15 ## Lesser General Public License for more details.
16 ##
17 ## You should have received a copy of the GNU Lesser General Public
18 ## License along with this program; if not, contact:
19 ##
20 ## Free Software Foundation Voice: +1-617-542-5942
21 ## 59 Temple Place - Suite 330 Fax: +1-617-542-2652
22 ## Boston, MA 02111-1307, USA gnu@gnu.org
23
24 """
25 statprof is intended to be a fairly simple statistical profiler for
26 python. It was ported directly from a statistical profiler for guile,
27 also named statprof, available from guile-lib [0].
28
29 [0] http://wingolog.org/software/guile-lib/statprof/
30
31 To start profiling, call statprof.start():
32 >>> start()
33
34 Then run whatever it is that you want to profile, for example:
35 >>> import test.pystone; test.pystone.pystones()
36
37 Then stop the profiling and print out the results:
38 >>> stop()
39 >>> display()
456e86a @bos Kill off trailing whitespace
authored
40 % cumulative self
41 time seconds seconds name
a85bd0c Initial commit
Jeff Muizelaar authored
42 26.72 1.40 0.37 pystone.py:79:Proc0
43 13.79 0.56 0.19 pystone.py:133:Proc1
44 13.79 0.19 0.19 pystone.py:208:Proc8
45 10.34 0.16 0.14 pystone.py:229:Func2
46 6.90 0.10 0.10 pystone.py:45:__init__
47 4.31 0.16 0.06 pystone.py:53:copy
48 ...
49
69b84b2 @bos Improve docs
authored
50 All of the numerical data is statistically approximate. In the
51 following column descriptions, and in all of statprof, "time" refers
52 to execution time (both user and system), not wall clock time.
a85bd0c Initial commit
Jeff Muizelaar authored
53
54 % time
55 The percent of the time spent inside the procedure itself (not
56 counting children).
57
58 cumulative seconds
59 The total number of seconds spent in the procedure, including
60 children.
61
62 self seconds
63 The total number of seconds spent in the procedure itself (not
64 counting children).
65
66 name
67 The name of the procedure.
68
69 By default statprof keeps the data collected from previous runs. If you
70 want to clear the collected data, call reset():
71 >>> reset()
72
fee07fa @bos Increase the default sampling frequency to 1000 Hz.
authored
73 reset() can also be used to change the sampling frequency from the
74 default of 1000 Hz. For example, to tell statprof to sample 50 times a
75 second:
a85bd0c Initial commit
Jeff Muizelaar authored
76 >>> reset(50)
77
78 This means that statprof will sample the call stack after every 1/50 of
79 a second of user + system time spent running on behalf of the python
80 process. When your process is idle (for example, blocking in a read(),
81 as is the case at the listener), the clock does not advance. For this
82 reason statprof is not currently not suitable for profiling io-bound
83 operations.
84
85 The profiler uses the hash of the code object itself to identify the
86 procedures, so it won't confuse different procedures with the same name.
87 They will show up as two different rows in the output.
88
89 Right now the profiler is quite simplistic. I cannot provide
90 call-graphs or other higher level information. What you see in the
91 table is pretty much all there is. Patches are welcome :-)
92
93
94 Threading
95 ---------
96
97 Because signals only get delivered to the main thread in Python,
98 statprof only profiles the main thread. However because the time
99 reporting function uses per-process timers, the results can be
100 significantly off if other threads' work patterns are not similar to the
101 main thread's work patterns.
102 """
103 from __future__ import division
104
105 import os
9ae3cc6 @bos Track line numbers properly.
authored
106 import signal
a85bd0c Initial commit
Jeff Muizelaar authored
107
7972fe8 @cyberdelia add profile contextmanager
cyberdelia authored
108 from contextlib import contextmanager
109
a85bd0c Initial commit
Jeff Muizelaar authored
110
7972fe8 @cyberdelia add profile contextmanager
cyberdelia authored
111 __all__ = ['start', 'stop', 'reset', 'display', 'profile']
a85bd0c Initial commit
Jeff Muizelaar authored
112
113
114 ###########################################################################
115 ## Utils
116
117 def clock():
118 times = os.times()
119 return times[0] + times[1]
120
121
122 ###########################################################################
123 ## Collection data structures
124
125 class ProfileState(object):
126 def __init__(self, frequency=None):
127 self.reset(frequency)
128
129 def reset(self, frequency=None):
130 # total so far
131 self.accumulated_time = 0.0
132 # start_time when timer is active
133 self.last_start_time = None
134 # total count of sampler calls
135 self.sample_count = 0
136 # a float
137 if frequency:
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
138 self.sample_interval = 1.0 / frequency
a85bd0c Initial commit
Jeff Muizelaar authored
139 elif not hasattr(self, 'sample_interval'):
fee07fa @bos Increase the default sampling frequency to 1000 Hz.
authored
140 # default to 1000 Hz
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
141 self.sample_interval = 1.0 / 1000.0
a85bd0c Initial commit
Jeff Muizelaar authored
142 else:
143 # leave the frequency as it was
144 pass
145 self.remaining_prof_time = None
146 # for user start/stop nesting
147 self.profile_level = 0
148 # whether to catch apply-frame
149 self.count_calls = False
150 # gc time between start() and stop()
151 self.gc_time_taken = 0
152
153 def accumulate_time(self, stop_time):
154 self.accumulated_time += stop_time - self.last_start_time
155
156 state = ProfileState()
157
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
158
9ae3cc6 @bos Track line numbers properly.
authored
159 class CodeKey(object):
160 cache = {}
161
162 __slots__ = ('filename', 'lineno', 'name')
163
164 def __init__(self, frame):
165 code = frame.f_code
a85bd0c Initial commit
Jeff Muizelaar authored
166 self.filename = code.co_filename
9ae3cc6 @bos Track line numbers properly.
authored
167 self.lineno = frame.f_lineno
168 self.name = code.co_name
169
170 def __eq__(self, other):
171 try:
172 return (self.lineno == other.lineno and
173 self.filename == other.filename)
174 except:
175 return False
176
177 def __hash__(self):
178 return hash((self.lineno, self.filename))
179
180 @classmethod
181 def get(cls, frame):
182 k = (frame.f_code.co_filename, frame.f_lineno)
183 try:
184 return cls.cache[k]
185 except KeyError:
186 v = cls(frame)
187 cls.cache[k] = v
188 return v
189
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
190
9ae3cc6 @bos Track line numbers properly.
authored
191 class CallData(object):
192 all_calls = {}
193
194 __slots__ = ('key', 'call_count', 'cum_sample_count', 'self_sample_count')
195
196 def __init__(self, key):
197 self.key = key
a85bd0c Initial commit
Jeff Muizelaar authored
198 self.call_count = 0
199 self.cum_sample_count = 0
200 self.self_sample_count = 0
201
9ae3cc6 @bos Track line numbers properly.
authored
202 @classmethod
203 def get(cls, key):
204 try:
205 return cls.all_calls[key]
206 except KeyError:
207 v = CallData(key)
208 cls.all_calls[key] = v
209 return v
a85bd0c Initial commit
Jeff Muizelaar authored
210
211
212 ###########################################################################
213 ## SIGPROF handler
214
215 def sample_stack_procs(frame):
216 state.sample_count += 1
9ae3cc6 @bos Track line numbers properly.
authored
217 key = CodeKey.get(frame)
218 CallData.get(key).self_sample_count += 1
a85bd0c Initial commit
Jeff Muizelaar authored
219
9ae3cc6 @bos Track line numbers properly.
authored
220 keys_seen = set()
a85bd0c Initial commit
Jeff Muizelaar authored
221 while frame:
9ae3cc6 @bos Track line numbers properly.
authored
222 key = CodeKey.get(frame)
223 keys_seen.add(key)
a85bd0c Initial commit
Jeff Muizelaar authored
224 frame = frame.f_back
9ae3cc6 @bos Track line numbers properly.
authored
225 for key in keys_seen:
226 CallData.get(key).cum_sample_count += 1
a85bd0c Initial commit
Jeff Muizelaar authored
227
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
228
a85bd0c Initial commit
Jeff Muizelaar authored
229 def profile_signal_handler(signum, frame):
230 if state.profile_level > 0:
231 state.accumulate_time(clock())
232 sample_stack_procs(frame)
233 signal.setitimer(signal.ITIMER_PROF,
234 state.sample_interval, 0.0)
235 state.last_start_time = clock()
236
237
238 ###########################################################################
239 ## Profiling API
240
241 def is_active():
242 return state.profile_level > 0
243
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
244
a85bd0c Initial commit
Jeff Muizelaar authored
245 def start():
69b84b2 @bos Improve docs
authored
246 '''Install the profiling signal handler, and start profiling.'''
a85bd0c Initial commit
Jeff Muizelaar authored
247 state.profile_level += 1
248 if state.profile_level == 1:
249 state.last_start_time = clock()
250 rpt = state.remaining_prof_time
251 state.remaining_prof_time = None
252 signal.signal(signal.SIGPROF, profile_signal_handler)
253 signal.setitimer(signal.ITIMER_PROF,
254 rpt or state.sample_interval, 0.0)
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
255 state.gc_time_taken = 0 # dunno
256
456e86a @bos Kill off trailing whitespace
authored
257
a85bd0c Initial commit
Jeff Muizelaar authored
258 def stop():
69b84b2 @bos Improve docs
authored
259 '''Stop profiling, and uninstall the profiling signal handler.'''
a85bd0c Initial commit
Jeff Muizelaar authored
260 state.profile_level -= 1
261 if state.profile_level == 0:
262 state.accumulate_time(clock())
263 state.last_start_time = None
264 rpt = signal.setitimer(signal.ITIMER_PROF, 0.0, 0.0)
265 signal.signal(signal.SIGPROF, signal.SIG_IGN)
266 state.remaining_prof_time = rpt[0]
f825408 @cyberdelia strict pep8 compliance
cyberdelia authored
267 state.gc_time_taken = 0 # dunno
268
456e86a @bos Kill off trailing whitespace
authored
269
a85bd0c Initial commit
Jeff Muizelaar authored
270 def reset(frequency=None):
69b84b2 @bos Improve docs
authored
271 '''Clear out the state of the profiler. Do not call while the
272 profiler is running.
273
274 The optional frequency argument specifies the number of samples to
275 collect per second.'''
a85bd0c Initial commit
Jeff Muizelaar authored
276 assert state.profile_level == 0, "Can't reset() while statprof is running"
9ae3cc6 @bos Track line numbers properly.
authored
277 CallData.all_calls.clear()
278 CodeKey.cache.clear()
a85bd0c Initial commit
Jeff Muizelaar authored
279 state.reset(frequency)
456e86a @bos Kill off trailing whitespace
authored
280
a85bd0c Initial commit
Jeff Muizelaar authored
281
7972fe8 @cyberdelia add profile contextmanager
cyberdelia authored
282 @contextmanager
283 def profile():
284 start()
285 try:
286 yield
287 finally:
288 stop()
289 display()
290
291
a85bd0c Initial commit
Jeff Muizelaar authored
292 ###########################################################################
293 ## Reporting API
294
295 class CallStats(object):
296 def __init__(self, call_data):
297 self_samples = call_data.self_sample_count
298 cum_samples = call_data.cum_sample_count
299 nsamples = state.sample_count
300 secs_per_sample = state.accumulated_time / nsamples
9ae3cc6 @bos Track line numbers properly.
authored
301 basename = os.path.basename(call_data.key.filename)
a85bd0c Initial commit
Jeff Muizelaar authored
302
9ae3cc6 @bos Track line numbers properly.
authored
303 self.name = '%s:%d:%s' % (basename, call_data.key.lineno,
304 call_data.key.name)
a85bd0c Initial commit
Jeff Muizelaar authored
305 self.pcnt_time_in_proc = self_samples / nsamples * 100
306 self.cum_secs_in_proc = cum_samples * secs_per_sample
307 self.self_secs_in_proc = self_samples * secs_per_sample
308 self.num_calls = None
309 self.self_secs_per_call = None
310 self.cum_secs_per_call = None
311
6015918 @bos Add support for display to non-stdout
authored
312 def display(self, fp):
313 print >> fp, ('%6.2f %9.2f %9.2f %s' % (self.pcnt_time_in_proc,
314 self.cum_secs_in_proc,
315 self.self_secs_in_proc,
316 self.name))
a85bd0c Initial commit
Jeff Muizelaar authored
317
318
6015918 @bos Add support for display to non-stdout
authored
319 def display(fp=None):
69b84b2 @bos Improve docs
authored
320 '''Print statistics, either to stdout or the given file object.'''
321
6015918 @bos Add support for display to non-stdout
authored
322 if fp is None:
e9f5e2e @jedbrown Need to import sys to use sys.stdout
jedbrown authored
323 import sys
6015918 @bos Add support for display to non-stdout
authored
324 fp = sys.stdout
a85bd0c Initial commit
Jeff Muizelaar authored
325 if state.sample_count == 0:
6015918 @bos Add support for display to non-stdout
authored
326 print >> fp, ('No samples recorded.')
a85bd0c Initial commit
Jeff Muizelaar authored
327 return
328
9ae3cc6 @bos Track line numbers properly.
authored
329 l = [CallStats(x) for x in CallData.all_calls.itervalues()]
a85bd0c Initial commit
Jeff Muizelaar authored
330 l.sort(reverse=True, key=lambda x: x.self_secs_in_proc)
331 l = [(x.self_secs_in_proc, x.cum_secs_in_proc, x) for x in l]
332 l = [x[2] for x in l]
333
6015918 @bos Add support for display to non-stdout
authored
334 print >> fp, ('%5.5s %10.10s %7.7s %-8.8s' %
335 ('% ', 'cumulative', 'self', ''))
456e86a @bos Kill off trailing whitespace
authored
336 print >> fp, ('%5.5s %9.9s %8.8s %-8.8s' %
6015918 @bos Add support for display to non-stdout
authored
337 ("time", "seconds", "seconds", "name"))
a85bd0c Initial commit
Jeff Muizelaar authored
338
339 for x in l:
6015918 @bos Add support for display to non-stdout
authored
340 x.display(fp)
a85bd0c Initial commit
Jeff Muizelaar authored
341
6015918 @bos Add support for display to non-stdout
authored
342 print >> fp, ('---')
343 print >> fp, ('Sample count: %d' % state.sample_count)
344 print >> fp, ('Total time: %f seconds' % state.accumulated_time)
Something went wrong with that request. Please try again.