Skip to content
This repository

Enabling layered storage (local,memcached,datastore) for your application. Keywords: GAE, Google App Engine

branch: master

Fetching latest commit…


Cannot retrieve the latest commit at this time

Octocat-spinner-32 PerformanceEngine
Octocat-spinner-32 tests
Octocat-spinner-32 LICENSE
Octocat-spinner-32 README.mdown

Performance Engine (v1.0)

What is it?

PerformanceEngine is a simple wrapper module that enables layered data model and query result storage on Google Application Engine.

More features coming soon!


  • Layered data storage (local,memcache or datastore).
  • Models that live in cache only (local or memcache).
  • Cached queries!
  • Lighweight (1 package, 2 files)
  • Seamless integration into existing projects (call pdb.put instead of db.put).
  • Different result types (list, key-model dict,name-model dict) to increase developer performance.
  • Built-in handlers for common errors. (DeadlineExceededError,CapabilityDisabledError)
  • Unified and simple API (if you figure out how to use pdb.get and pdb.put, you're set for good)


What is it good for?

  • Applications that have high amount and frequency of write operations.
  • Applications that do batch data processing.
  • Applications that like to stay under free quota of App Engine.

What is it not good for?

  • Applications that do lots of transactions with highly sensitive data.
  • Applications that rely heavily on queries rather than get and put operations.

Basic Usage

If no additional parameters are given, PerformanceEngine works as memcache supported datastore. A get() function will try to get entities from memcache first, failing that, it tries to retrieve from datastore. It will then write entities that are fetched from datastore into memcache.

models = [model1,model2..]
keys = pdb.put()
#Key for an entity that's in datastore but not in memcache

#model1 & model2 is retrieved from memcache but model for
# new_db_key is retrieved from datastore. It is also saved 
#into memcache for future calls.


Using different storage layers:

from PerformanceEngine import pdb

model1 = SomeModel(key_name='model1')
model2 = SomeModel(key_name='model2')
model3 = SomeModel(key_name='model3')

#Write to local cache

#Write to both memcache and local cache

#Write to all layers

#Read from local cache
model1 = SomeModel.get_by_key_name('model1',_storage='local')

#Read from memcache 
model2 = SomeModel.get_by_key_name('model2',_storage='memcache')

#Read from datastore (same as db.get)
model3 = SomeModel.get_by_key_name('model3',_storage='datastore')

Advanced Usage

Please see wiki.


There aren't any example projects (yet) but PerformanceEngine is heavily used in TweetHit.

Known Bugs

  • If you try to write models without key_names into cache only without writing them to datastore first, it will fail as they won't have any keys created for them.


This module is released under the BSD License. You can find the full text of the license in the LICENSE file.


Something went wrong with that request. Please try again.