Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Riak-backed virtual filesystem (proof of concept)
Branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.


riakfs - riak-backed virtual filesystem

A `pyfilesystem`_ subclass that uses `Riak`_ for distributed storage.

The idea is to abuse the ``MemoryFS`` class to get a tree-of-objects
representation of a filesystem, but where file objects are stored
transparently in a `Riak`_ cluster. The tree-of-objects is itself
serializable to json and, after a call to the filesystem's ``close()``
(or ``save()``) method, is stored as the filesystem "journal".


Virtualenv and pip is recommended, I had problems installing `protobuf` with

In a virtualenv, install the following::

    pip install fs
    pip install protobuf
    pip install riak

There is no **riakfs** release yet, so install the development version::

    pip install -e git+ssh://

Move to the riakfs source directory and, assuming that you don't have a Riak
cluster installed, run the ``riak.devstart`` make target::

    make riak.devstart

This will download and install Riak and create and start a three node
development cluster. The installation assumes that you have Erlang installed -
the `Basho Wiki`_ has details (a source install on Ubuntu was straightforward).

Check that the cluster is up with::

    ps aux | grep beam

And run the riakfs tests with::

    make test

which assumes `pytest`_ is installed and that a Riak server is running
on port 8091.


Create a RiakFS object::

    >>> from riakfs import RiakFS
    >>> fs = RiakFS('test-bucket')

Delete anything previously saved::

    >>> fs.reset()

There's a delay here while Riak does its thing and, for the sake of not having
a failed doctest, wait a few seconds::

    >>> import time
    >>> time.sleep(3)
    >>> bucket = fs.bucket
    >>> bucket.get_keys()

Create a directory::

    >>> fs.tree(terminal_colors=False)
    >>> fs.makedir('a')
    >>> fs.tree(terminal_colors=False)
    `-- a
    >>> fs.makedir('a/b')
    >>> fs.tree(terminal_colors=False)
    `-- a
        `-- b

Directories aren't saved as objects, so the only key present is the special
"journal" key - __VFS__. The journal has been autosaved as a result of the
``makedir`` call::

    >>> time.sleep(3)
    >>> bucket.get_keys()

Create a file::

    >>> fs.setcontents('a/b/test1.txt', 'testingtestingtesting')
    >>> fs.tree(terminal_colors=False)
    `-- a
        `-- b
            `-- test1.txt

Files autosave on close::

    >>> time.sleep(3)
    >>> sorted(bucket.get_keys())
    [u'__VFS__', u'__VFS__/a/b/test1.txt']

Retrieve the file directly from Riak::

    >>> obj = bucket.get_binary('__VFS__/a/b/test1.txt')
    >>> print(obj.get_data())

.. _Riak:
.. _Basho Wiki:

Something went wrong with that request. Please try again.