Skip to content
This repository

A ubiquitous mini-profiler for Google App Engine, inspired by mvc-mini-profiler

branch: master

Fix occasional crash in the sampling profiler

Summary:
Before this commit, when the sampling profiler was finished, the main thread
would set a flag for the sampling thread to stop profiling but wouldn't wait for
the background thread to actually exit. This meant that the background thread
would sometimes add one last sample to the list of samples while we were in the
middle of processing that list. The Profile.results function makes two passes
over the list of samples and assumes that they'll be the same both times, so it
was crashing when the list got modified in the middle.

The fix is to wait for the sampling thread to exit before processing the
results.

Test Plan:
Temporarily put a sleep(1) just before the crash point and a sleep(.01) at the
top of the top of the sampling loop. With these sleeps, the home page
consistently crashes with sampling profiling enabled. Adding the join statement
fixes this crash.

Reviewers: ben

Reviewed By: ben

Subscribers: alpert

Differential Revision: http://phabricator.khanacademy.org/D7621
latest commit bfba282d56
Alan Pierce alangpierce authored
Octocat-spinner-32 static Update stack trace while moving sampling slider March 11, 2014
Octocat-spinner-32 templates Upgrade jQuery version for shared profiler page February 14, 2014
Octocat-spinner-32 unformatter Improved error handling for parsing errors January 24, 2013
Octocat-spinner-32 .arcconfig Fix mini profiler's redirect when redirecting to URLs w/ hash fragments. October 02, 2013
Octocat-spinner-32 .gitignore switch to allow all admin users to see profiler tab. June 27, 2011
Octocat-spinner-32 README.md Another README tweak to test push hook January 23, 2013
Octocat-spinner-32 __init__.py Initial import June 26, 2011
Octocat-spinner-32 _line_profiler.so Add line-by-line profiling option November 07, 2013
Octocat-spinner-32 appstats_profiler.py Don't spam logs when prettification fails July 03, 2013
Octocat-spinner-32 cleanup.py Add new sampling profiler and make threadsafe for python27/multithrea… January 03, 2013
Octocat-spinner-32 config.py Let users use appengine_config to override should_profile settings January 04, 2013
Octocat-spinner-32 cookies.py Let gae_mini_profiler users disable w/ a little UI on/off setting in … November 09, 2011
Octocat-spinner-32 instrumented_profiler.py Add ability to download raw profiling data January 07, 2014
Octocat-spinner-32 line_profiler.py Add line-by-line profiling option November 07, 2013
Octocat-spinner-32 linebyline_profiler.py Search sys.meta_path for PathRestrictingImportHook March 10, 2014
Octocat-spinner-32 main.py Add ability to download raw profiling data January 07, 2014
Octocat-spinner-32 profiler.py Add color coding for long waits and add icon for likely duplicate ite… January 29, 2014
Octocat-spinner-32 sampling_profiler.py Fix occasional crash in the sampling profiler March 23, 2014
Octocat-spinner-32 templatetags.py Don't specify gae_mini_profiler import prefix for now. TODO: figure o… January 04, 2013
Octocat-spinner-32 util.py Add new sampling profiler and make threadsafe for python27/multithrea… January 03, 2013
README.md

Google App Engine Mini Profiler

gae_mini_profiler is a quick drop-in WSGI app that provides ubiquitous profiling of your existing GAE projects. It exposes RPC statistics and CPU profiling output for users of your choosing on your production site. Only requests coming from users of your choosing will be profiled, and others will not suffer any performance degradation, so you can use this profiler to learn about production performance without stressing about slowing users down. See screenshots and features below.

This project is heavily inspired by the Stack Exchange team's impressive mini-profiler.

See it in action

Play around with a demo App Engine applications with gae_mini_profiler enabled at http://mini-profiler.appspot.com.

Screenshots

All profiled pages have total milliseconds in corner, which can be expanded...


...to show more details...


...about remote procedure call performance...


...or CPU profiler output.


Choose between an instrumented CPU profiler (above) or a sampling profiler (below).


Ajax requests and redirects are also profiled and added to the corner of your page.


Any Python logging module output is also available for easy access.

Getting Started

  1. Download this repository's source and copy the gae_mini_profiler/ folder into your App Engine project's root directory.

  2. Add the following two handler definitions to app.yaml:

    handlers:
    - url: /gae_mini_profiler/static
      static_dir: gae_mini_profiler/static
    - url: /gae_mini_profiler/.*
      script: gae_mini_profiler.main.application
    
  3. Modify the WSGI application you want to profile by wrapping it with the gae_mini_profiler WSGI application.

    import gae_mini_profiler.profiler
    ...
    application = webapp.WSGIApplication([...])
    application = gae_mini_profiler.profiler.ProfilerWSGIMiddleware(application)
    
  4. Modify your template to include our javascript and stylesheets just before your ending body tag.

    There is a profiler_includes() function in gae_mini_profiler.templatetags that spits out the right code for these scripts and stylesheets.

    Using any template engine of your choice, call this function at the end of your template:

            ...
            {% profiler_includes %}
        </body>
    </html>
    

    Note that these resources will not be loaded on requests when the profiler is disabled, so you don't need to worry about extra HTTP requests slowing down your users.

    Using Django? You can register a simple_tag to expose this to your templates:

    register = template.create_template_register()
    @register.simple_tag
    def profiler_includes():
        return gae_mini_profiler.templatetags.profiler_includes()
    

    Using jinja2? You can expose this function to your templates easily:

    webapp2_extras.jinja2.default_config = {
        "globals": {
            "profiler_includes": gae_mini_profiler.templatetags.profiler_includes
        }
    }
    

    Using anything else to generate your HTML? Just find some way to spit the results of profiler_includes() into your HTML. Doesn't have to be anything fancy.

  5. You're all set! Now you just need to choose when you want to enable the profiler by overriding a simple function. By default it's enabled on the dev server and disabled in production. To enable it for App Engine admins in production, add the following to appengine_config.py:

    def gae_mini_profiler_should_profile_production():
            from google.appengine.api import users
            return users.is_current_user_admin()
    

    In appengine_config.py you can override both of the following...

    def gae_mini_profiler_should_profile_production(): pass
    def gae_mini_profiler_should_profile_development(): pass
    

    ...with any logic you want to choose when the profiler should be enabled.

Features

  • Production profiling without impacting normal users
  • Easily profile all requests, including ajax calls
  • Summaries of RPC call types and their performance so you can quickly figure out whether datastore, memcache, or urlfetch is your bottleneck
  • Redirect chains are tracked -- quickly examine the profile of not just the currently rendered request, but any preceding request that issued a 302 redirect leading to the current page.
  • Share individual profile results with others by sending link
  • Duplicate RPC calls are flagged for easy spotting in case you're repeating memcache or datastore queries
  • Choose from either an instrumented or sampling CPU profiler to quickly figure out where your requests are spending time

Dependencies

  • jQuery must be included somewhere on your page.
  • (Optional) If you want the fancy slider selector for the Logs output, jQuery UI must also be included with its Slider plugin.

Bonus

gae_mini_profiler is currently in production use at Khan Academy. If you make good use of it elsewhere, please lemme know.

FAQ

  1. What's the license? MIT licensed.
  2. I had my appstats_RECORD_FRACTION variable set to 0.1, which means only 10% of my queries were getting profiles generated. This meant that most of the time gae_mini_profiler was failing with a javascript error, because the appstats variable was null.

    If you are using appengine_config.py to customize Appstats behavior you should add this to the top of your "appstats_should_record" method.

    def appstats_should_record(env):
        from gae_mini_profiler.config import should_profile
        if should_profile(env):
            return True
    
Something went wrong with that request. Please try again.