Skip to content

Latest commit

 

History

History
201 lines (142 loc) · 5.81 KB

index.rst

File metadata and controls

201 lines (142 loc) · 5.81 KB
Raw

simple key-value storage api

simplekv is an API for key-value store of binary data. Due to its basic interface, it is easy to implemented a large number of backends. simplekv's origins are in storing user-uploaded files on websites, but its low overhead and design should make it applicable for numerous other problems, an example is a session backend for the Flask framework.

Built upon the solid foundation are a few optional bells and whistles, such as automatic ID generation/hashing (in simplekv.idgen). A number of backends are available, ranging from ~.FilesystemStore to support for Amazon S3 and Google Storage through ~.BotoStore.

A faster in-memory store suitable for session management and caching is supported through ~.RedisStore

Example

Here's a simple example:

from simplekv.fs import FilesystemStore

store = FilesystemStore('./data')

store.put(u'key1', 'hello')

# will print "hello"
print store.get(u'key1')

# move the contents of a file to "key2" as efficiently as possible
store.put_file(u'key2', '/path/to/data')

Note that by changing the first two lines to:

from simplekv.memory.redisstore import RedisStore
import redis

store = RedisStore(redis.StrictRedis())

you could use the code exactly the same way, this time storing data inside a Redis database.

Why you should use simplekv

no server dependencies

simplekv does only depend on python and possibly a few libraries easily fetchable from PyPI, if you want to use extra features. You do not have to run and install any server software to use simplekv (but can at any point later on).

specializes in (even large!) blobs

The fastest, most basic simplekv backend implementation stores files on your harddrive and is just as fast. This underlines the focus on storing big blobs without overhead or metadata. A typical usecase is starting out small with local files and then migrating all your binary data to something like Amazon's S3.

Table of contents

filesystem boto azure gcstorage gae memory db git idgen crypt decorators cache development

changes

The core API

simplekv.KeyValueStore

Some backends support an efficient copy operation, which is provided by a mixin class:

simplekv.CopyMixin

In addition to that, a mixin class is available for backends that provide a method to support URL generation:

simplekv.UrlMixin

simplekv.UrlKeyValueStore

Some backends support setting a time-to-live on keys for automatic expiration, this is represented by the ~simplekv.TimeToLiveMixin:

simplekv.TimeToLiveMixin

simplekv.TimeToLiveMixin.put

simplekv.TimeToLiveMixin.put_file

default_ttl_secs = simplekv.NOT_SET

Passing None for any time-to-live parameter will cause this value to be used.

simplekv.TimeToLiveMixin.ttl_support

simplekv.VALID_KEY_REGEXP

simplekv.VALID_KEY_RE

Implementing a new backend

Subclassing ~simplekv.KeyValueStore is the fastest way to implement a new backend. It suffices to override the ~simplekv.KeyValueStore._delete, ~simplekv.KeyValueStore.iter_keys, ~simplekv.KeyValueStore._open and ~simplekv.KeyValueStore._put_file methods, as all the other methods have default implementations that call these.

After that, you can override any number of underscore-prefixed methods with more specialized implementations to gain speed improvements.

Default implementation

Classes derived from ~simplekv.KeyValueStore inherit a number of default implementations for the core API methods. Specifically, the ~simplekv.KeyValueStore.delete, ~simplekv.KeyValueStore.get, ~simplekv.KeyValueStore.get_file, ~simplekv.KeyValueStore.keys, ~simplekv.KeyValueStore.open, ~simplekv.KeyValueStore.put, ~simplekv.KeyValueStore.put_file, methods will each call the ~simplekv.KeyValueStore._check_valid_key method if a key has been provided and then call one of the following protected methods:

simplekv.KeyValueStore._check_valid_key

simplekv.KeyValueStore._delete

simplekv.KeyValueStore._get

simplekv.KeyValueStore._get_file

simplekv.KeyValueStore._get_filename

simplekv.KeyValueStore._has_key

simplekv.KeyValueStore._open

simplekv.KeyValueStore._put

simplekv.KeyValueStore._put_file

simplekv.KeyValueStore._put_filename

Atomicity

Every call to a method on a KeyValueStore results in a single operation on the underlying backend. No guarantees are made above that, if you check if a key exists and then try to retrieve it, it may have already been deleted in between (instead, retrieve and catch the exception).

Python 3

All of the examples are written in Python 2. However, Python 3 is fully supported and tested. When using simplekv in a Python 3 environment, the only important thing to remember is that keys are always strings and values are always byte-objects.

Indices and tables

  • genindex
  • modindex
  • search