A ubiquitous mini-profiler for Google App Engine, inspired by mvc-mini-profiler
JavaScript CSS Python HTML
Latest commit 62401c6 Aug 23, 2016 @csilvers csilvers Support memcache chunking in the sampling profiler.
Summary:
Now we'll be able to show sampled profile results even if the
pickled+compressed output is >1M.

Test Plan:
Deployed a znd that used this new code, and used the sampling profiler
on the homepage of a new request to see how long it took
get_path_table() to run (I deleted the `@layer_cache` for
get_path_table() first).  I saw the mini-profiler dropdown appear,
whereas before it didn't, and could visualize the profile.

Reviewers: chris, benkraft

Reviewed By: benkraft

Differential Revision: https://phabricator.khanacademy.org/D30226
Failed to load latest commit information.
static Catch requests made with `fetch` and show profiler data about them Aug 5, 2016
templates Merge branch 'lipis-patch-1' Sep 25, 2014
unformatter Improved error handling for parsing errors Jan 24, 2013
.arcconfig Update this to use the new SSL phabricator. Nov 26, 2014
.arclint Fix the regexp for arc lint to correctly only match a single line. Oct 13, 2015
.gitignore switch to allow all admin users to see profiler tab. Jun 27, 2011
README.md Reformat README Dec 9, 2015
__init__.py Initial import Jun 26, 2011
_line_profiler.so Add line-by-line profiling option Nov 7, 2013
appstats_profiler.py Ignore GetSystemStatsRequest in duplicate RPC detection. Jun 24, 2014
cleanup.py Add new sampling profiler and make threadsafe for python27/multithrea… Jan 4, 2013
config.py Merge branch 'bslatkin-master' Sep 25, 2014
cookies.py Let gae_mini_profiler users disable w/ a little UI on/off setting in … Nov 10, 2011
instrumented_profiler.py Add ability to download raw profiling data Jan 7, 2014
line_profiler.py Add line-by-line profiling option Nov 7, 2013
linebyline_profiler.py Merge branch 'bslatkin-master' Sep 25, 2014
main.py Add link to .cpuprofile to GAE Mini Profiler. Aug 2, 2014
profiler.py Support memcache chunking in the sampling profiler. Aug 23, 2016
sampling_profiler.py Allow passing a sleep_fxn, similar to time_fxn Apr 19, 2016
templatetags.py Don't specify gae_mini_profiler import prefix for now. TODO: figure o… Jan 4, 2013
util.py Merge branch 'bslatkin-master' Sep 25, 2014

README.md

Google App Engine Mini Profiler

gae_mini_profiler is a quick drop-in WSGI app that provides ubiquitous profiling of your existing GAE projects. It exposes RPC statistics and CPU profiling output for users of your choosing on your production site. Only requests coming from users of your choosing will be profiled, and others will not suffer any performance degradation, so you can use this profiler to learn about production performance without stressing about slowing users down. See screenshots and features below.

This project is heavily inspired by the Stack Exchange team's impressive mini-profiler.

See it in action

Play around with a demo App Engine applications with gae_mini_profiler enabled at http://mini-profiler.appspot.com.

Screenshots

All profiled pages have total milliseconds in corner, which can be expanded...

...to show more details...

...about remote procedure call performance...

...or CPU profiler output.

Choose between an instrumented CPU profiler (above) or a sampling profiler (below).

Ajax requests and redirects are also profiled and added to the corner of your page.

Any Python logging module output is also available for easy access.

Getting Started

  1. Download this repository's source and copy the gae_mini_profiler/ folder into your App Engine project's root directory.

  2. Add the following two handler definitions to app.yaml:

    handlers:
    - url: /gae_mini_profiler/static
      static_dir: gae_mini_profiler/static
    - url: /gae_mini_profiler/.*
      script: gae_mini_profiler.main.application
  3. Modify the WSGI application you want to profile by wrapping it with the gae_mini_profiler WSGI application.

    import gae_mini_profiler.profiler
    ...
    application = webapp.WSGIApplication([...])
    application = gae_mini_profiler.profiler.ProfilerWSGIMiddleware(application)
  4. Modify your template to include our javascript and stylesheets just before your ending body tag.

    There is a profiler_includes() function in gae_mini_profiler.templatetags that spits out the right code for these scripts and stylesheets.

    Using any template engine of your choice? Call this function at the end of your template:

            ...
            {% profiler_includes %}
        </body>
    </html>

    Note that these resources will not be loaded on requests when the profiler is disabled, so you don't need to worry about extra HTTP requests slowing down your users.

    Using Django? You can register a simple_tag to expose this to your templates:

    register = template.create_template_register()
    @register.simple_tag
    def profiler_includes():
        return gae_mini_profiler.templatetags.profiler_includes()

    Using Jinja2? You can expose this function to your templates easily:

    webapp2_extras.jinja2.default_config = {
        "globals": {
            "profiler_includes": gae_mini_profiler.templatetags.profiler_includes
        }
    }

    Using anything else to generate your HTML? Just find some way to spit the results of profiler_includes() into your HTML. Doesn't have to be anything fancy.

  5. You're all set! Now you just need to choose when you want to enable the profiler by overriding a simple function. By default it's enabled on the dev server and disabled in production. To enable it for App Engine admins in production, add the following to appengine_config.py:

    def gae_mini_profiler_should_profile_production():
        from google.appengine.api import users
        return users.is_current_user_admin()

    In appengine_config.py you can override both of the following...

    def gae_mini_profiler_should_profile_production(): pass
    def gae_mini_profiler_should_profile_development(): pass

    ...with any logic you want to choose when the profiler should be enabled.

Features

  • Production profiling without impacting normal users
  • Easily profile all requests, including ajax calls
  • Summaries of RPC call types and their performance so you can quickly figure out whether datastore, memcache, or urlfetch is your bottleneck
  • Redirect chains are tracked -- quickly examine the profile of not just the currently rendered request, but any preceding request that issued a 302 redirect leading to the current page.
  • Share individual profile results with others by sending link
  • Duplicate RPC calls are flagged for easy spotting in case you're repeating memcache or datastore queries
  • Choose from either an instrumented or sampling CPU profiler to quickly figure out where your requests are spending time

Dependencies

  • jQuery must be included somewhere on your page.
  • (Optional) If you want the fancy slider selector for the Logs output, jQuery UI must also be included with its Slider plugin.

Bonus

gae_mini_profiler is currently in production use at Khan Academy. If you make good use of it elsewhere, please lemme know.

FAQ

  1. What's the license? MIT licensed. There is also code from the Chromium project's DevTools whose license is at static/chrome/inspector/devtools.html.
  2. I had my appstats_RECORD_FRACTION variable set to 0.1, which means only 10% of my queries were getting profiles generated. This meant that most of the time gae_mini_profiler was failing with a javascript error, because the appstats variable was null.
  3. If you are using appengine_config.py to customize Appstats behavior you should add this to the top of your appstats_should_record method.

    def appstats_should_record(env):
        from gae_mini_profiler.config import should_profile
        if should_profile(env):
            return True