Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 299 lines (235 sloc) 10.135 kb
a85bd0c Initial commit
Jeff Muizelaar authored
1 ## statprof.py
2 ## Copyright (C) 2011 Alex Fraser <alex at phatcore dot com>
3 ## Copyright (C) 2004,2005 Andy Wingo <wingo at pobox dot com>
4 ## Copyright (C) 2001 Rob Browning <rlb at defaultvalue dot org>
5
6 ## This library is free software; you can redistribute it and/or
7 ## modify it under the terms of the GNU Lesser General Public
8 ## License as published by the Free Software Foundation; either
9 ## version 2.1 of the License, or (at your option) any later version.
10 ##
11 ## This library is distributed in the hope that it will be useful,
12 ## but WITHOUT ANY WARRANTY; without even the implied warranty of
13 ## MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
14 ## Lesser General Public License for more details.
15 ##
16 ## You should have received a copy of the GNU Lesser General Public
17 ## License along with this program; if not, contact:
18 ##
19 ## Free Software Foundation Voice: +1-617-542-5942
20 ## 59 Temple Place - Suite 330 Fax: +1-617-542-2652
21 ## Boston, MA 02111-1307, USA gnu@gnu.org
22
23 """
24 statprof is intended to be a fairly simple statistical profiler for
25 python. It was ported directly from a statistical profiler for guile,
26 also named statprof, available from guile-lib [0].
27
28 [0] http://wingolog.org/software/guile-lib/statprof/
29
30 To start profiling, call statprof.start():
31 >>> start()
32
33 Then run whatever it is that you want to profile, for example:
34 >>> import test.pystone; test.pystone.pystones()
35
36 Then stop the profiling and print out the results:
37 >>> stop()
38 >>> display()
39 % cumulative self
40 time seconds seconds name
41 26.72 1.40 0.37 pystone.py:79:Proc0
42 13.79 0.56 0.19 pystone.py:133:Proc1
43 13.79 0.19 0.19 pystone.py:208:Proc8
44 10.34 0.16 0.14 pystone.py:229:Func2
45 6.90 0.10 0.10 pystone.py:45:__init__
46 4.31 0.16 0.06 pystone.py:53:copy
47 ...
48
49 All of the numerical data with the exception of the calls column is
50 statistically approximate. In the following column descriptions, and
51 in all of statprof, "time" refers to execution time (both user and
52 system), not wall clock time.
53
54 % time
55 The percent of the time spent inside the procedure itself (not
56 counting children).
57
58 cumulative seconds
59 The total number of seconds spent in the procedure, including
60 children.
61
62 self seconds
63 The total number of seconds spent in the procedure itself (not
64 counting children).
65
66 name
67 The name of the procedure.
68
69 By default statprof keeps the data collected from previous runs. If you
70 want to clear the collected data, call reset():
71 >>> reset()
72
73 reset() can also be used to change the sampling frequency. For example,
74 to tell statprof to sample 50 times a second:
75 >>> reset(50)
76
77 This means that statprof will sample the call stack after every 1/50 of
78 a second of user + system time spent running on behalf of the python
79 process. When your process is idle (for example, blocking in a read(),
80 as is the case at the listener), the clock does not advance. For this
81 reason statprof is not currently not suitable for profiling io-bound
82 operations.
83
84 The profiler uses the hash of the code object itself to identify the
85 procedures, so it won't confuse different procedures with the same name.
86 They will show up as two different rows in the output.
87
88 Right now the profiler is quite simplistic. I cannot provide
89 call-graphs or other higher level information. What you see in the
90 table is pretty much all there is. Patches are welcome :-)
91
92
93 Threading
94 ---------
95
96 Because signals only get delivered to the main thread in Python,
97 statprof only profiles the main thread. However because the time
98 reporting function uses per-process timers, the results can be
99 significantly off if other threads' work patterns are not similar to the
100 main thread's work patterns.
101
102
103 Implementation notes
104 --------------------
105
106 The profiler works by setting the unix profiling signal ITIMER_PROF to
107 go off after the interval you define in the call to reset(). When the
108 signal fires, a sampling routine is run which looks at the current
109 procedure that's executing, and then crawls up the stack, and for each
110 frame encountered, increments that frame's code object's sample count.
111 Note that if a procedure is encountered multiple times on a given stack,
112 it is only counted once. After the sampling is complete, the profiler
113 resets profiling timer to fire again after the appropriate interval.
114
115 Meanwhile, the profiler keeps track, via os.times(), how much CPU time
116 (system and user -- which is also what ITIMER_PROF tracks), has elapsed
117 while code has been executing within a start()/stop() block.
118
119 The profiler also tries to avoid counting or timing its own code as
120 much as possible.
121 """
122
123
124 from __future__ import division
125
126 import signal
127 import os
128
129
130 __all__ = ['start', 'stop', 'reset', 'display']
131
132
133 ###########################################################################
134 ## Utils
135
136 def clock():
137 times = os.times()
138 return times[0] + times[1]
139
140
141 ###########################################################################
142 ## Collection data structures
143
144 class ProfileState(object):
145 def __init__(self, frequency=None):
146 self.reset(frequency)
147
148 def reset(self, frequency=None):
149 # total so far
150 self.accumulated_time = 0.0
151 # start_time when timer is active
152 self.last_start_time = None
153 # total count of sampler calls
154 self.sample_count = 0
155 # a float
156 if frequency:
157 self.sample_interval = 1.0/frequency
158 elif not hasattr(self, 'sample_interval'):
159 # default to 100 Hz
160 self.sample_interval = 1.0/100.0
161 else:
162 # leave the frequency as it was
163 pass
164 self.remaining_prof_time = None
165 # for user start/stop nesting
166 self.profile_level = 0
167 # whether to catch apply-frame
168 self.count_calls = False
169 # gc time between start() and stop()
170 self.gc_time_taken = 0
171
172 def accumulate_time(self, stop_time):
173 self.accumulated_time += stop_time - self.last_start_time
174
175 state = ProfileState()
176
177 ## call_data := { code object: CallData }
178 call_data = {}
179 class CallData(object):
180 def __init__(self, code):
181 self.name = code.co_name
182 self.filename = code.co_filename
183 self.lineno = code.co_firstlineno
184 self.call_count = 0
185 self.cum_sample_count = 0
186 self.self_sample_count = 0
187 call_data[code] = self
188
189 def get_call_data(code):
190 return call_data.get(code, None) or CallData(code)
191
192
193 ###########################################################################
194 ## SIGPROF handler
195
196 def sample_stack_procs(frame):
197 state.sample_count += 1
198 get_call_data(frame.f_code).self_sample_count += 1
199
200 code_seen = {}
201 while frame:
202 code_seen[frame.f_code] = True
203 frame = frame.f_back
204 for code in code_seen.keys():
205 get_call_data(code).cum_sample_count += 1
206
207 def profile_signal_handler(signum, frame):
208 if state.profile_level > 0:
209 state.accumulate_time(clock())
210 sample_stack_procs(frame)
211 signal.setitimer(signal.ITIMER_PROF,
212 state.sample_interval, 0.0)
213 state.last_start_time = clock()
214
215
216 ###########################################################################
217 ## Profiling API
218
219 def is_active():
220 return state.profile_level > 0
221
222 def start():
223 state.profile_level += 1
224 if state.profile_level == 1:
225 state.last_start_time = clock()
226 rpt = state.remaining_prof_time
227 state.remaining_prof_time = None
228 signal.signal(signal.SIGPROF, profile_signal_handler)
229 signal.setitimer(signal.ITIMER_PROF,
230 rpt or state.sample_interval, 0.0)
231 state.gc_time_taken = 0 # dunno
232
233 def stop():
234 state.profile_level -= 1
235 if state.profile_level == 0:
236 state.accumulate_time(clock())
237 state.last_start_time = None
238 rpt = signal.setitimer(signal.ITIMER_PROF, 0.0, 0.0)
239 signal.signal(signal.SIGPROF, signal.SIG_IGN)
240 state.remaining_prof_time = rpt[0]
241 state.gc_time_taken = 0 # dunno
242
243 def reset(frequency=None):
244 assert state.profile_level == 0, "Can't reset() while statprof is running"
245 call_data.clear()
246 state.reset(frequency)
247
248
249 ###########################################################################
250 ## Reporting API
251
252 class CallStats(object):
253 def __init__(self, call_data):
254 self_samples = call_data.self_sample_count
255 cum_samples = call_data.cum_sample_count
256 nsamples = state.sample_count
257 secs_per_sample = state.accumulated_time / nsamples
258 basename = os.path.basename(call_data.filename)
259
260 self.name = '%s:%d:%s' % (basename, call_data.lineno, call_data.name)
261 self.pcnt_time_in_proc = self_samples / nsamples * 100
262 self.cum_secs_in_proc = cum_samples * secs_per_sample
263 self.self_secs_in_proc = self_samples * secs_per_sample
264 self.num_calls = None
265 self.self_secs_per_call = None
266 self.cum_secs_per_call = None
267
6015918 @bos Add support for display to non-stdout
authored
268 def display(self, fp):
269 print >> fp, ('%6.2f %9.2f %9.2f %s' % (self.pcnt_time_in_proc,
270 self.cum_secs_in_proc,
271 self.self_secs_in_proc,
272 self.name))
a85bd0c Initial commit
Jeff Muizelaar authored
273
274
6015918 @bos Add support for display to non-stdout
authored
275 def display(fp=None):
276 if fp is None:
277 fp = sys.stdout
a85bd0c Initial commit
Jeff Muizelaar authored
278 if state.sample_count == 0:
6015918 @bos Add support for display to non-stdout
authored
279 print >> fp, ('No samples recorded.')
a85bd0c Initial commit
Jeff Muizelaar authored
280 return
281
282 l = [CallStats(x) for x in call_data.values()]
283 l.sort(reverse=True, key=lambda x: x.self_secs_in_proc)
284 l = [(x.self_secs_in_proc, x.cum_secs_in_proc, x) for x in l]
285 l = [x[2] for x in l]
286
6015918 @bos Add support for display to non-stdout
authored
287 print >> fp, ('%5.5s %10.10s %7.7s %-8.8s' %
288 ('% ', 'cumulative', 'self', ''))
289 print >> fp, ('%5.5s %9.9s %8.8s %-8.8s' %
290 ("time", "seconds", "seconds", "name"))
a85bd0c Initial commit
Jeff Muizelaar authored
291
292 for x in l:
6015918 @bos Add support for display to non-stdout
authored
293 x.display(fp)
a85bd0c Initial commit
Jeff Muizelaar authored
294
6015918 @bos Add support for display to non-stdout
authored
295 print >> fp, ('---')
296 print >> fp, ('Sample count: %d' % state.sample_count)
297 print >> fp, ('Total time: %f seconds' % state.accumulated_time)
a85bd0c Initial commit
Jeff Muizelaar authored
298
Something went wrong with that request. Please try again.