A simple, unified Python API for accessing multiple data stores like databases or file system directories asynchronously
py-datastore is a generic layer of abstraction for data store and database access. It is a simple API with the aim to enable application development in a datastore-agnostic way, allowing datastores to be swapped seamlessly without changing application code. Thus, one can leverage different datastores with different strengths without committing the application to one datastore throughout its lifetime. It looks like this:
+---------------+
| application | <--- No cumbersome SQL or Mongo specific queries!
+---------------+
| <--- simple datastore API calls
+---------------+
| datastore | <--- datastore implementation for underlying db
+---------------+
| <--- database specific calls
+---------------+
| various dbs | <--- MySQL, Redis, MongoDB, FS, …
+---------------+
In addition, datastore adapters and collections significantly simplify creating interesting data access patterns such as caching, sharding and tiering.
From pypi (WARNING – This is currently outdated!):
pip install datastore
From source:
git clone https://github.com/ipfs/py-datastore.git
cd datastore
flit install
Only the filesystem and the in-memory DictDatastore are included in this project.
For other known implementations please see the links below.
NOTE: Struck through entries are not compatible with the latest py-datastore releases!
datastore.aws - aws s3 implementationdatastore.git - git implementationdatastore.mongo - monogdb implementationdatastore.memcached - memcached implementationdatastore.pylru - pylru cache implementationdatastore.redis - redis implementationdatastore.leveldb - leveldb implementation
TBD! Some outdated examples from the previous iteration of this library:
>>> import datastore.core
>>> ds = datastore.DictDatastore()
>>>
>>> hello = datastore.Key('hello')
>>> ds.put(hello, 'world')
>>> ds.contains(hello)
True
>>> ds.get(hello)
'world'
>>> ds.delete(hello)
>>> ds.get(hello)
None
>>> import datastore.filesystem
>>>
>>> ds = datastore.filesystem.FileSystemDatastore('/tmp/.test_datastore')
>>>
>>> hello = datastore.Key('hello')
>>> ds.put(hello, 'world')
>>> ds.contains(hello)
True
>>> ds.get(hello)
'world'
>>> ds.delete(hello)
>>> ds.get(hello)
None
>>> import pymongo
>>> import datastore.core
>>>
>>> from datastore.mongo import MongoDatastore
>>> from datastore.pylru import LRUCacheDatastore
>>> from datastore.filesystem import FileSystemDatastore
>>>
>>> conn = pymongo.Connection()
>>> mongo = MongoDatastore(conn.test_db)
>>>
>>> cache = LRUCacheDatastore(1000)
>>> fs = FileSystemDatastore('/tmp/.test_db')
>>>
>>> ds = datastore.TieredDatastore([cache, mongo, fs])
>>>
>>> hello = datastore.Key('hello')
>>> ds.put(hello, 'world')
>>> ds.contains(hello)
True
>>> ds.get(hello)
'world'
>>> ds.delete(hello)
>>> ds.get(hello)
None
>>> import datastore.core
>>>
>>> shards = [datastore.DictDatastore() for i in range(0, 10)]
>>>
>>> ds = datastore.ShardedDatastore(shards)
>>>
>>> hello = datastore.Key('hello')
>>> ds.put(hello, 'world')
>>> ds.contains(hello)
True
>>> ds.get(hello)
'world'
>>> ds.delete(hello)
>>> ds.get(hello)
None
The documentation can be found at (WARNING – This is currently outdated!): http://datastore.readthedocs.org/en/latest/
The datastore API places an emphasis on simplicity and elegance. Only four core methods must be implemented (get, put, delete, query). There are two variants of this API: object datastores and binary datastores. Object datastores store streams of rich objects (like database rows or JSON objects), binary datastores just store abitrary blobs of binary data (like filesystem files).
Object datastores may also impose additional limitations on the types and shape of objects that may be stored. It is also not an error if the “object stream” may only ever consist of a single object value.
Return the binary or object data stream named by key or raise KeyError if there
is no such value.
Arguments
---------
key
Key naming the binary or object data to retrieve
Raises
------
KeyError
The given key was not present in this datastore
RuntimeError
An internal error occurred
Store the object or binary data (stream) value into key. If the key does
not yet exist it will be created.
Arguments
---------
key
Key naming the binary or object data slot to store at
value
A list or stream of objects or bytes to store
Raises
------
RuntimeError
An internal error occurred
Remove the object or binary data named by key. If it did not exist KeyError
will be raised.
Arguments
---------
key
Key naming the object or binary data slot to remove
Raises
------
KeyError
The given key was not present in this datastore
RuntimeError
An internal error occurred
Return an iterable of objects matching criteria expressed in query.
Only object datastores that are able to perform optimized queries
should implement this method. If a query is considered too complex
for your datastore implementation, raise NotImplementedError rather
then falling back to linear search.
Arguments
---------
query
Object describing which objects to match and return
Raises
------
RuntimeError
An internal error occurred
Datastore implementors are free to implement specialized features, pertinent only to a subset of datastores, with the understanding that these should aim for generality and will most likely not be implemented across other datastores.
When implementing such features, please remember the goal of this project: A simple, unified API for multiple data stores. When making heavy use of a particular library's specific functionality, perhaps one should not use datastore and should directly use that library.
A Key represents the unique identifier of an object.
Our Key scheme is inspired by classic Unix file systems.
Keys are meant to be unique across a system. Keys are hierarchical, incorporating increasingly specific namespaces. Thus keys can be deemed children or ancestors of other keys.
Key('/Comedy')
Key('/Comedy/MontyPython')
Currently active maintainers:
Past maintainers:
We'd like to thank the following contributers for their help (in order of appearance):
For bug reports, feature requests and other issues please visit the GitHub issue tracker.
Code contributions are primarily accepted through GitHub PRs. The general flow for submitting contributions is:
- Sign in to GitHub.
- Fork the repository by clicking on the “Fork” button on the top-right of the repository page (you will have to switch to desktop mode on mobile).
- In your forked repository page click on the big green “Clone or download” button on the right-hand side and copy the SSH URL.
- On your system run:
git clone <COPIED_SSH_URL>to download the repository. - Make changes to your repository locally and commit them using
git add <CHANGED_FILES>, thengit commit. (If you have several related changes in mind please make one commit for each of them.)- Please also observe the next paragrah on things you should check before commiting!
- Push them to your repository using
git push. - In your repository page click on the “New pull request” button.
- Review the proposed changes and continue with “Create Pull Request”.
- Enter a suitable title and description for your changes and confirm by pressing “Create Pull Request” again.
When commiting code please run the tests first and code checkers first and try to fix all problems you observe:
- Run tests using:
tox -e py3XwhereXis your Python 3 minor version (ie:7for Python 3.7) - Check compliance with the code style:
tox -e styleck - Check for static typing problems:
tox -e typeck - Also note that all code should be documented!
Code in this repository follows the PEP 8 conventions for styling Python code with the exception of using tabs for indentation rather then spaces. In general, code that passes the style checker will be considered acceptable in this regard, so try to fix all issues reported by the style checker first before proposing a code change.
Please also be aware that all interactions related to IPFS, libp2p and Multiformats are subject to the IPFS Code of Conduct.
Please write and contribute implementations for other data stores. This project can only be complete with lots of help.
py-datastore released under the MIT License.
py-datastore was orignally written by Juan Batiz-Benet. It was originally part of py-dronestore before being rewritten in 2011.