Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Add line-by-line profiling option

Summary:
This ports the functionality I originally wrote for
flask_debugtoolbar_lineprofilerpanel to work with the mini profiler.

{F6708}

See the original here: https://github.com/phleet/flask_debugtoolbar_lineprofilerpanel

Coded in Oslo, Norway.

Test Plan:
With this patch applied, point your browser to your site with the mini profiler
enabled, open up one of the request profiles, then under "settings", choose
"line-by-line" as the CPU profiling option.

Reload the page. Opening up the profile details, you should be presented with
a message saying:
```
To mark functions for profiling, put this in the source of the file containing
the function you want to profile:

    from gae_mini_profiler import linebyline_profiler
    linebyline_profiler.line_profile(SomeClass.some_instance_method)
    linebyline_profiler.line_profile(some_other_function)
```

In KA's case, the profiler needs to be imported through third_party, so the
actual import becomes `from third_party.gae_mini_profiler import
linebyline_profiler`.

Choose a function that gets executed 1 or more times when loading the profiler,
then follow the above instructions to mark it for profiling. Reload your
browser.

Opening up the profile details now, you should see a table presenting the
line-by-line results.

Go back to your code and remove the profiling code. Reload the browser, and the
details should revert back to the message explaining how to mark functions.

Reviewers: ben

Reviewed By: ben

CC: alpert, chris

Differential Revision: http://phabricator.khanacademy.org/D4819
  • Loading branch information...
commit fe6b7389e03c5537b34ea854bdacedb02f95fa1a 1 parent 304e19a
@jlfwong jlfwong authored
View
BIN  _line_profiler.so
Binary file not shown
View
347 line_profiler.py
@@ -0,0 +1,347 @@
+#!/usr/bin/env python
+# -*- coding: UTF-8 -*-
+
+import cPickle
+from cStringIO import StringIO
+import inspect
+import linecache
+import optparse
+import os
+import sys
+
+from _line_profiler import LineProfiler as CLineProfiler
+
+
+CO_GENERATOR = 0x0020
+def is_generator(f):
+ """ Return True if a function is a generator.
+ """
+ isgen = (f.func_code.co_flags & CO_GENERATOR) != 0
+ return isgen
+
+# Code to exec inside of LineProfiler.__call__ to support PEP-342-style
+# generators in Python 2.5+.
+pep342_gen_wrapper = '''
+def wrap_generator(self, func):
+ """ Wrap a generator to profile it.
+ """
+ def f(*args, **kwds):
+ g = func(*args, **kwds)
+ # The first iterate will not be a .send()
+ self.enable_by_count()
+ try:
+ item = g.next()
+ finally:
+ self.disable_by_count()
+ input = (yield item)
+ # But any following one might be.
+ while True:
+ self.enable_by_count()
+ try:
+ item = g.send(input)
+ finally:
+ self.disable_by_count()
+ input = (yield item)
+ return f
+'''
+
+class LineProfiler(CLineProfiler):
+ """ A profiler that records the execution times of individual lines.
+ """
+
+ def __call__(self, func):
+ """ Decorate a function to start the profiler on function entry and stop
+ it on function exit.
+ """
+ self.add_function(func)
+ if is_generator(func):
+ f = self.wrap_generator(func)
+ else:
+ f = self.wrap_function(func)
+ f.__module__ = func.__module__
+ f.__name__ = func.__name__
+ f.__doc__ = func.__doc__
+ f.__dict__.update(getattr(func, '__dict__', {}))
+ return f
+
+ if sys.version_info[:2] >= (2,5):
+ # Delay compilation because the syntax is not compatible with older
+ # Python versions.
+ exec pep342_gen_wrapper
+ else:
+ def wrap_generator(self, func):
+ """ Wrap a generator to profile it.
+ """
+ def f(*args, **kwds):
+ g = func(*args, **kwds)
+ while True:
+ self.enable_by_count()
+ try:
+ item = g.next()
+ finally:
+ self.disable_by_count()
+ yield item
+ return f
+
+ def wrap_function(self, func):
+ """ Wrap a function to profile it.
+ """
+ def f(*args, **kwds):
+ self.enable_by_count()
+ try:
+ result = func(*args, **kwds)
+ finally:
+ self.disable_by_count()
+ return result
+ return f
+
+ def dump_stats(self, filename):
+ """ Dump a representation of the data to a file as a pickled LineStats
+ object from `get_stats()`.
+ """
+ lstats = self.get_stats()
+ f = open(filename, 'wb')
+ try:
+ cPickle.dump(lstats, f, cPickle.HIGHEST_PROTOCOL)
+ finally:
+ f.close()
+
+ def print_stats(self, stream=None):
+ """ Show the gathered statistics.
+ """
+ lstats = self.get_stats()
+ show_text(lstats.timings, lstats.unit, stream=stream)
+
+ def run(self, cmd):
+ """ Profile a single executable statment in the main namespace.
+ """
+ import __main__
+ dict = __main__.__dict__
+ return self.runctx(cmd, dict, dict)
+
+ def runctx(self, cmd, globals, locals):
+ """ Profile a single executable statement in the given namespaces.
+ """
+ self.enable_by_count()
+ try:
+ exec cmd in globals, locals
+ finally:
+ self.disable_by_count()
+ return self
+
+ def runcall(self, func, *args, **kw):
+ """ Profile a single function call.
+ """
+ self.enable_by_count()
+ try:
+ return func(*args, **kw)
+ finally:
+ self.disable_by_count()
+
+
+def show_func(filename, start_lineno, func_name, timings, unit, stream=None):
+ """ Show results for a single function.
+ """
+ if stream is None:
+ stream = sys.stdout
+ print >>stream, "File: %s" % filename
+ print >>stream, "Function: %s at line %s" % (func_name, start_lineno)
+ template = '%6s %9s %12s %8s %8s %-s'
+ d = {}
+ total_time = 0.0
+ linenos = []
+ for lineno, nhits, time in timings:
+ total_time += time
+ linenos.append(lineno)
+ print >>stream, "Total time: %g s" % (total_time * unit)
+ if not os.path.exists(filename):
+ print >>stream, ""
+ print >>stream, "Could not find file %s" % filename
+ print >>stream, "Are you sure you are running this program from the same directory"
+ print >>stream, "that you ran the profiler from?"
+ print >>stream, "Continuing without the function's contents."
+ # Fake empty lines so we can see the timings, if not the code.
+ nlines = max(linenos) - min(min(linenos), start_lineno) + 1
+ sublines = [''] * nlines
+ else:
+ all_lines = linecache.getlines(filename)
+ sublines = inspect.getblock(all_lines[start_lineno-1:])
+ for lineno, nhits, time in timings:
+ d[lineno] = (nhits, time, '%5.1f' % (float(time) / nhits),
+ '%5.1f' % (100*time / total_time))
+ linenos = range(start_lineno, start_lineno + len(sublines))
+ empty = ('', '', '', '')
+ header = template % ('Line #', 'Hits', 'Time', 'Per Hit', '% Time',
+ 'Line Contents')
+ print >>stream, ""
+ print >>stream, header
+ print >>stream, '=' * len(header)
+ for lineno, line in zip(linenos, sublines):
+ nhits, time, per_hit, percent = d.get(lineno, empty)
+ print >>stream, template % (lineno, nhits, time, per_hit, percent,
+ line.rstrip('\n').rstrip('\r'))
+ print >>stream, ""
+
+def show_text(stats, unit, stream=None):
+ """ Show text for the given timings.
+ """
+ if stream is None:
+ stream = sys.stdout
+ print >>stream, 'Timer unit: %g s' % unit
+ print >>stream, ''
+ for (fn, lineno, name), timings in sorted(stats.items()):
+ show_func(fn, lineno, name, stats[fn, lineno, name], unit, stream=stream)
+
+# A %lprun magic for IPython.
+def magic_lprun(self, parameter_s=''):
+ """ Execute a statement under the line-by-line profiler from the
+ line_profiler module.
+
+ Usage:
+ %lprun -f func1 -f func2 <statement>
+
+ The given statement (which doesn't require quote marks) is run via the
+ LineProfiler. Profiling is enabled for the functions specified by the -f
+ options. The statistics will be shown side-by-side with the code through the
+ pager once the statement has completed.
+
+ Options:
+
+ -f <function>: LineProfiler only profiles functions and methods it is told
+ to profile. This option tells the profiler about these functions. Multiple
+ -f options may be used. The argument may be any expression that gives
+ a Python function or method object. However, one must be careful to avoid
+ spaces that may confuse the option parser. Additionally, functions defined
+ in the interpreter at the In[] prompt or via %run currently cannot be
+ displayed. Write these functions out to a separate file and import them.
+
+ One or more -f options are required to get any useful results.
+
+ -D <filename>: dump the raw statistics out to a pickle file on disk. The
+ usual extension for this is ".lprof". These statistics may be viewed later
+ by running line_profiler.py as a script.
+
+ -T <filename>: dump the text-formatted statistics with the code side-by-side
+ out to a text file.
+
+ -r: return the LineProfiler object after it has completed profiling.
+ """
+ # Local imports to avoid hard dependency.
+ from distutils.version import LooseVersion
+ import IPython
+ ipython_version = LooseVersion(IPython.__version__)
+ if ipython_version < '0.11':
+ from IPython.genutils import page
+ from IPython.ipstruct import Struct
+ from IPython.ipapi import UsageError
+ else:
+ from IPython.core.page import page
+ from IPython.utils.ipstruct import Struct
+ from IPython.core.error import UsageError
+
+ # Escape quote markers.
+ opts_def = Struct(D=[''], T=[''], f=[])
+ parameter_s = parameter_s.replace('"',r'\"').replace("'",r"\'")
+ opts, arg_str = self.parse_options(parameter_s, 'rf:D:T:', list_all=True)
+ opts.merge(opts_def)
+
+ global_ns = self.shell.user_global_ns
+ local_ns = self.shell.user_ns
+
+ # Get the requested functions.
+ funcs = []
+ for name in opts.f:
+ try:
+ funcs.append(eval(name, global_ns, local_ns))
+ except Exception, e:
+ raise UsageError('Could not find function %r.\n%s: %s' % (name,
+ e.__class__.__name__, e))
+
+ profile = LineProfiler(*funcs)
+
+ # Add the profiler to the builtins for @profile.
+ import __builtin__
+ if 'profile' in __builtin__.__dict__:
+ had_profile = True
+ old_profile = __builtin__.__dict__['profile']
+ else:
+ had_profile = False
+ old_profile = None
+ __builtin__.__dict__['profile'] = profile
+
+ try:
+ try:
+ profile.runctx(arg_str, global_ns, local_ns)
+ message = ''
+ except SystemExit:
+ message = """*** SystemExit exception caught in code being profiled."""
+ except KeyboardInterrupt:
+ message = ("*** KeyboardInterrupt exception caught in code being "
+ "profiled.")
+ finally:
+ if had_profile:
+ __builtin__.__dict__['profile'] = old_profile
+
+ # Trap text output.
+ stdout_trap = StringIO()
+ profile.print_stats(stdout_trap)
+ output = stdout_trap.getvalue()
+ output = output.rstrip()
+
+ if ipython_version < '0.11':
+ page(output, screen_lines=self.shell.rc.screen_length)
+ else:
+ page(output)
+ print message,
+
+ dump_file = opts.D[0]
+ if dump_file:
+ profile.dump_stats(dump_file)
+ print '\n*** Profile stats pickled to file',\
+ `dump_file`+'.',message
+
+ text_file = opts.T[0]
+ if text_file:
+ pfile = open(text_file, 'w')
+ pfile.write(output)
+ pfile.close()
+ print '\n*** Profile printout saved to text file',\
+ `text_file`+'.',message
+
+ return_value = None
+ if opts.has_key('r'):
+ return_value = profile
+
+ return return_value
+
+
+def load_ipython_extension(ip):
+ """ API for IPython to recognize this module as an IPython extension.
+ """
+ ip.define_magic('lprun', magic_lprun)
+
+
+def load_stats(filename):
+ """ Utility function to load a pickled LineStats object from a given
+ filename.
+ """
+ f = open(filename, 'rb')
+ try:
+ lstats = cPickle.load(f)
+ finally:
+ f.close()
+ return lstats
+
+
+def main():
+ usage = "usage: %prog profile.lprof"
+ parser = optparse.OptionParser(usage=usage, version='%prog 1.0b2')
+
+ options, args = parser.parse_args()
+ if len(args) != 1:
+ parser.error("Must provide a filename.")
+ lstats = load_stats(args[0])
+ show_text(lstats.timings, lstats.unit)
+
+if __name__ == '__main__':
+ main()
View
157 linebyline_profiler.py
@@ -0,0 +1,157 @@
+"""CPU profiler that works by collecting line-by-line stats.
+
+This works by storing a list of functions to profile, then telling
+the third party line_profiler module to profile those functions.
+"""
+
+import collections
+import inspect
+import linecache
+import os
+import re
+import sys
+
+_is_dev_server = os.environ["SERVER_SOFTWARE"].startswith("Devel")
+
+# We can't use LineProfiler in production because it requires a C-extension,
+# but we can monkey-patch it in here for use on the dev server:
+if _is_dev_server:
+ # white-list the line_profiler C extension
+ sys.meta_path[3]._enabled_regexes.append(re.compile(r'.*line_profiler.*'))
+
+ import line_profiler
+ assert line_profiler # Silence pyflakes
+else:
+ line_profiler = None
+
+_FUNCTION_MARKER = "__gae_linebyline_profile"
+
+_functions_to_profile = []
+
+
+def line_profile(f):
+ """The passed function will be included in the line profile displayed by
+ the line profiler panel.
+ """
+ # TODO(jlfwong): See if this is needed.
+ f.__dict__[_FUNCTION_MARKER] = True
+ if f not in _functions_to_profile:
+ _functions_to_profile.append(f)
+
+ return f
+
+
+def _process_line_stats(line_stats):
+ """Convert line_profiler.LineStats instance into a dict.
+
+ The returned dict has the following format:
+
+ [{
+ "filename": the filename of the function being profiled
+ "start_lineno": the first line number of the function
+ "func_name": the name of the function
+ "total_time_ms": total time spent inside the function in ms
+ "total_time_ms_s": formatted string version of above
+ "timings": [{
+ 'lineno': line number being profiled
+ 'line': string source line being profiled
+ 'perc_time': percent of total time spent on this line
+ 'perc_time_s': formatted string version of above
+ 'time_ms': total time spent on this line
+ 'time_ms_s': formatted string version of above
+ 'numhits': the number of times this line was run
+ }, ...]
+ }, ...]
+ """
+
+ profile_results = []
+
+ if not line_stats:
+ return profile_results
+
+ # We want timings in ms (instead of CPython's microseconds)
+ multiplier = line_stats.unit / 1e-3
+
+ for key, timings in sorted(line_stats.timings.items()):
+ if not timings:
+ continue
+
+ filename, start_lineno, func_name = key
+
+ all_lines = linecache.getlines(filename)
+ sublines = inspect.getblock(all_lines[start_lineno-1:])
+ end_lineno = start_lineno + len(sublines)
+
+ line_to_timing = collections.defaultdict(lambda: (-1, 0))
+
+ for (lineno, nhits, time) in timings:
+ line_to_timing[lineno] = (nhits, time)
+
+ padded_timings = []
+
+ for lineno in range(start_lineno, end_lineno):
+ nhits, time = line_to_timing[lineno]
+ padded_timings.append( (lineno, nhits, time) )
+
+ timings = []
+
+ result = {
+ 'filename': filename,
+ 'start_lineno': start_lineno,
+ 'func_name': func_name,
+ 'total_time_ms': (sum([time for _, _, time in padded_timings]) *
+ multiplier),
+ 'timings': []
+ }
+
+ result['total_time_ms_s'] = '%.0f' % result['total_time_ms']
+
+ for (lineno, nhits, time) in padded_timings:
+ time_ms = time * multiplier
+ perc_time = (100.0 * time_ms) / result['total_time_ms']
+
+ result['timings'].append({
+ 'lineno': lineno,
+ 'line': all_lines[lineno - 1],
+ 'perc_time': perc_time,
+ 'perc_time_s': '%.1f' % perc_time,
+ 'time_ms': time_ms,
+ 'time_ms_s': "%.2f" % time_ms,
+ 'numhits': nhits
+ })
+
+ profile_results.append(result)
+
+ return profile_results
+
+
+class Profile(object):
+ """Profiler wrapping line_profiler."""
+ def __init__(self):
+ self.num_functions_marked = len(_functions_to_profile)
+
+ if line_profiler is None:
+ self.line_prof = None
+ else:
+ self.line_prof = line_profiler.LineProfiler()
+
+ for f in _functions_to_profile:
+ self.line_prof.add_function(f)
+
+ def results(self):
+ res = {
+ "is_dev_server": _is_dev_server,
+ "num_functions_marked": self.num_functions_marked,
+ "calls": []
+ }
+
+ if self.line_prof and self.num_functions_marked:
+ res["calls"] = _process_line_stats(self.line_prof.get_stats())
+
+ return res
+
+ def run(self, fxn):
+ if self.line_prof is None:
+ return fxn()
+ else:
+ return self.line_prof.runcall(fxn)
View
32 profiler.py
@@ -78,9 +78,11 @@ class Mode(object):
SIMPLE = "simple" # Simple start/end timing for the request as a whole
CPU_INSTRUMENTED = "instrumented" # Profile all function calls
CPU_SAMPLING = "sampling" # Sample call stacks
+ CPU_LINEBYLINE = "linebyline" # Line-by-line profiling on a subset of functions
RPC_ONLY = "rpc" # Profile all RPC calls
RPC_AND_CPU_INSTRUMENTED = "rpc_instrumented" # RPCs and all fxn calls
RPC_AND_CPU_SAMPLING = "rpc_sampling" # RPCs and sample call stacks
+ RPC_AND_CPU_LINEBYLINE = "rpc_linebyline" # RPCs and line-by-line profiling
@staticmethod
def get_mode(environ):
@@ -95,9 +97,11 @@ def get_mode(environ):
Mode.SIMPLE,
Mode.CPU_INSTRUMENTED,
Mode.CPU_SAMPLING,
+ Mode.CPU_LINEBYLINE,
Mode.RPC_ONLY,
Mode.RPC_AND_CPU_INSTRUMENTED,
- Mode.RPC_AND_CPU_SAMPLING]):
+ Mode.RPC_AND_CPU_SAMPLING,
+ Mode.RPC_AND_CPU_LINEBYLINE]):
mode = Mode.RPC_AND_CPU_INSTRUMENTED
return mode
@@ -107,19 +111,25 @@ def is_rpc_enabled(mode):
return mode in [
Mode.RPC_ONLY,
Mode.RPC_AND_CPU_INSTRUMENTED,
- Mode.RPC_AND_CPU_SAMPLING];
+ Mode.RPC_AND_CPU_SAMPLING]
@staticmethod
def is_sampling_enabled(mode):
return mode in [
Mode.CPU_SAMPLING,
- Mode.RPC_AND_CPU_SAMPLING];
+ Mode.RPC_AND_CPU_SAMPLING]
@staticmethod
def is_instrumented_enabled(mode):
return mode in [
Mode.CPU_INSTRUMENTED,
- Mode.RPC_AND_CPU_INSTRUMENTED];
+ Mode.RPC_AND_CPU_INSTRUMENTED]
+
+ @staticmethod
+ def is_linebyline_enabled(mode):
+ return mode in [
+ Mode.CPU_LINEBYLINE,
+ Mode.RPC_AND_CPU_LINEBYLINE]
class SharedStatsHandler(RequestHandler):
@@ -299,6 +309,7 @@ def __init__(self, request_id, mode):
self.mode = mode
self.instrumented_prof = None
self.sampling_prof = None
+ self.linebyline_prof = None
self.appstats_prof = None
self.temporary_redirect = False
self.handler = None
@@ -321,6 +332,8 @@ def profiler_results(self):
results.update(self.instrumented_prof.results())
elif self.sampling_prof:
results.update(self.sampling_prof.results())
+ elif self.linebyline_prof:
+ results.update(self.linebyline_prof.results())
return results
@@ -378,7 +391,7 @@ def profile_start_response(self, app, environ, start_response):
# Note that we don't import appstats_profiler at the top of
# this file so we don't bring in a lot of imports for users who
# don't have the profiler enabled.
- from gae_mini_profiler import appstats_profiler
+ from . import appstats_profiler
self.appstats_prof = appstats_profiler.Profile()
app = self.appstats_prof.wrap(app)
@@ -395,16 +408,21 @@ def profile_start_response(self, app, environ, start_response):
# Note that we don't import sampling_profiler at the top of
# this file so we don't bring in a lot of imports for users who
# don't have the profiler enabled.
- from gae_mini_profiler import sampling_profiler
+ from . import sampling_profiler
self.sampling_prof = sampling_profiler.Profile()
result_fxn_wrapper = self.sampling_prof.run
+ elif Mode.is_linebyline_enabled(self.mode):
+ from . import linebyline_profiler
+ self.linebyline_prof = linebyline_profiler.Profile()
+ result_fxn_wrapper = self.linebyline_prof.run
+
elif Mode.is_instrumented_enabled(self.mode):
# Turn on cProfile instrumented profiling for this request
# Note that we don't import instrumented_profiler at the top of
# this file so we don't bring in a lot of imports for users who
# don't have the profiler enabled.
- from gae_mini_profiler import instrumented_profiler
+ from . import instrumented_profiler
self.instrumented_prof = instrumented_profiler.Profile()
result_fxn_wrapper = self.instrumented_prof.run
View
2  sampling_profiler.py
@@ -26,7 +26,7 @@
import threading
import traceback
-from gae_mini_profiler import util
+from . import util
_is_dev_server = os.environ["SERVER_SOFTWARE"].startswith("Devel")
View
22 static/css/profiler.css
@@ -152,6 +152,28 @@
color: #CCC;
}
+.g-m-p .details table tr.linebyline-gt-10 {
+ background-color: #f33;
+}
+
+.g-m-p .details table tr.linebyline-gt-1 {
+ background-color: #faa;
+}
+
+.g-m-p .details h3 {
+ margin-top: 12px;
+ margin-bottom: 6px;
+}
+
+.g-m-p .details h4 {
+ margin-bottom: 6px;
+}
+
+.g-m-p code {
+ display: block;
+ font-family: monospace;
+}
+
.g-m-p .details th, .g-m-p .details th.header {
font-size: 12px;
background-color: #F7F7F7;
View
20 static/js/profiler.js
@@ -9,9 +9,11 @@ var GaeMiniProfiler = {
SIMPLE: "simple",
CPU_INSTRUMENTED: "instrumented",
CPU_SAMPLING: "sampling",
+ CPU_LINEBYLINE: "linebyline",
RPC_ONLY: "rpc",
RPC_AND_CPU_INSTRUMENTED: "rpc_instrumented",
- RPC_AND_CPU_SAMPLING: "rpc_sampling"
+ RPC_AND_CPU_SAMPLING: "rpc_sampling",
+ RPC_AND_CPU_LINEBYLINE: "rpc_linebyline"
},
init: function(requestId, fShowImmediately) {
@@ -104,11 +106,20 @@ var GaeMiniProfiler = {
},
/**
+ * True if profiler mode has enabled CPU line-by-line profiling
+ */
+ isLineByLineEnabled: function(mode) {
+ return (mode == this.modes.CPU_LINEBYLINE ||
+ mode == this.modes.RPC_AND_CPU_LINEBYLINE);
+ },
+
+ /**
* True if either CPU instrumentation or CPU sampling is enabled
*/
isCpuEnabled: function(mode) {
return (GaeMiniProfiler.isInstrumentedEnabled(mode) ||
- GaeMiniProfiler.isSamplingEnabled(mode));
+ GaeMiniProfiler.isSamplingEnabled(mode) ||
+ GaeMiniProfiler.isLineByLineEnabled(mode));
},
appendRedirectIds: function(requestId, queryString) {
@@ -324,9 +335,10 @@ var GaeMiniProfiler = {
var cpuSelector = "#cpu_disabled";
if (this.isInstrumentedEnabled(mode)) {
cpuSelector = "#cpu_instrumented";
- }
- else if (this.isSamplingEnabled(mode)) {
+ } else if (this.isSamplingEnabled(mode)) {
cpuSelector = "#cpu_sampling";
+ } else if (this.isLineByLineEnabled(mode)) {
+ cpuSelector = "#cpu_linebyline";
}
var rpcSelector = "#rpc_disabled";
View
77 static/js/template.tmpl
@@ -44,6 +44,7 @@
<td>
<input type="radio" id="cpu_instrumented" name="cpu" value="instrumented"/><label for="cpu_instrumented"> instrumented</label><br>
<input type="radio" id="cpu_sampling" name="cpu" value="sampling"/><label for="cpu_sampling"> sampling</label><br>
+ <input type="radio" id="cpu_linebyline" name="cpu" value="linebyline"/><label for="cpu_linebyline"> line-by-line</label><br>
<input type="radio" id="cpu_disabled" name="cpu" value=""/><label for="cpu_disabled"> disabled</label>
</td>
<td>
@@ -52,7 +53,7 @@
</td>
</tr>
<tr class="tips">
- <td>CPU profiling either keeps track of all function calls and their timings (instrumented) or periodically examines the call stack to figure out in which functions time is being spent during a request (sampling).</td>
+ <td>CPU profiling either keeps track of all function calls and their timings (instrumented), periodically examines the call stack to figure out in which functions time is being spent during a request (sampling), or tracks only specific functions (line-by-line).</td>
<td>RPC profiling monitors all remote procedure calls (think datastore queries, memcache accesses, and URL fetches) and their timings.</td>
</tr>
</table>
@@ -86,8 +87,10 @@
<div class="summary">
{{if GaeMiniProfiler.isInstrumentedEnabled(mode)}}
${profiler_results.total_time} <span class="ms">ms</span> spent in ${profiler_results.total_call_count} function calls
- {{else}}
+ {{else GaeMiniProfiler.isSamplingEnabled(mode)}}
${profiler_results.total_samples} sampled stack trace{{if profiler_results.total_samples != 1}}s{{/if}}
+ {{else GaeMiniProfiler.isLineByLineEnabled(mode)}}
+ ${profiler_results.num_functions_marked} profiled function{{if profiler_results.num_functions_marked != 1}}s{{/if}}
{{/if}}
</div>
</div>
@@ -138,7 +141,7 @@
{{/each}}
</table>
- {{else}}
+ {{else GaeMiniProfiler.isSamplingEnabled(mode)}}
{{if profiler_results.is_dev_server}}
<span class="warn">
@@ -171,6 +174,74 @@
{{/if}}
+ {{if GaeMiniProfiler.isLineByLineEnabled(mode)}}
+ {{if !profiler_results.is_dev_server}}
+ <span class="warn">
+ The line-by-line profiler can only be used in development.
+ </span>
+ {{else profiler_results.num_functions_marked == 0}}
+ <span>
+ To mark functions for profiling, put this in the source of the
+ file containing the function you want to profile:
+ </span>
+ <code>
+ from gae_mini_profiler.linebyline_profiler import line_profile<br>
+ <br>
+ # You can use it as a decorator...<br>
+ class SomeClass(object):<br>
+ &nbsp;&nbsp;&nbsp;&nbsp;@line_profile<br>
+ &nbsp;&nbsp;&nbsp;&nbsp;def some_instance_method(self):<br>
+ &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;self.cool_stuff()<br>
+ <br>
+ # ...or regular function call<br>
+ line_profile(SomeClass.some_instance_method)<br>
+ line_profile(some_other_function)
+ </code>
+ {{/if}}
+
+ {{each(i, fn_result) profiler_results.calls}}
+
+ <h3>${fn_result.func_name}</h3>
+ <h4>${fn_result.filename}:${fn_result.start_lineno}</h4>
+
+ <span>
+ Total Time: ${fn_result.total_time_ms_s} ms
+ </span>
+ <table>
+ <thead>
+ <th>&nbsp;</th>
+ <th>&nbsp;</th>
+ <th><nobr>% Time</nobr></th>
+ <th><nobr>Time (ms)</nobr></th>
+ <th>Hits</th>
+ </thead>
+ <tbody>
+ {{each(j, timing) fn_result.timings}}
+ {{if timing.perc_time > 10}}
+ <tr class="linebyline-gt-10">
+ {{else timing.perc_time > 1}}
+ <tr class="linebyline-gt-1">
+ {{else}}
+ <tr>
+ {{/if}}
+ <td>${timing.lineno}</td>
+ <td><pre>${timing.line}</pre></td>
+ {{if timing.numhits != -1}}
+ <td>${timing.perc_time_s}</td>
+ <td>${timing.time_ms_s}</td>
+ <td>${timing.numhits}</td>
+ {{else}}
+ <td>&nbsp;</td>
+ <td>&nbsp;</td>
+ <td>&nbsp;</td>
+ {{/if}}
+ </tr>
+ {{/each}}
+ </tbody>
+ </table>
+
+ {{/each}}
+ {{/if}}
</div>
{{/if}}

4 comments on commit fe6b738

@cdman

Unfortunately this includes a binary component which only works on MacOSX (at least it is compiled for both x86 and x64 :smile:). I added some guard code around it so that it doesn't die if the library can't be loaded: https://github.com/udacity/gae_mini_profiler/blob/master/line_profiler.py#L13 (and perhaps you could document how to get the given library? I tried installing line_profiler from pip, but that didn't work - I assume because of the GAE SDK import restrictions. I also tried to copy the _line_profiler.so installed by pip into the current directory but that didn't work either)

Edit: copying _line_profiler.so from your pip installation works (I copied it to the wrong directory initailly :p)

@jlfwong
Collaborator

@cdman Crud! That's my bad. This is what happens when the author and reviewer run the same OS :)

I'll work on getting a reasonable general solution out in the next week or so. It'll probably look something like this:

try:
    from _line_profiler_osx import LineProfiler as CLineProfiler
except ImportError:
    from _line_profiler_local import LineProfiler as CLineProfiler

Then add _line_profiler_local.so to the .gitignore.

Then inside the first party linebyline_profiler.py, I'll catch the ImportError and provide instructions to copy the .so out of a pip installation. Does that sound reasonable?

@cdman

I would recommend having a final fallback solution, like in the linked source code. Trouble is, if the code fails hard (with an exception), the entire page fails to load (and you don't have an easy way of fixing your mistake - ie. changing to a different profiling method).

@jlfwong
Collaborator

Finally got around to try to fix this: #69

Please sign in to comment.
Something went wrong with that request. Please try again.