Thu Dec 24 07:16:53 IST 2009             Abhishek Mishra <ideamonk at gmail.com>

                         -=========================-
                         |   PoorMan's DataStore   |
                         -=========================-

Preface/Bullshit -
------------------

    Yes I'm a Freetard, and my crime is of having a leecher's mentality. Maybe
    I don't have the same attitude towards a penny in my hand, or anything that
    has to do with the real world. But when it comes to something that is
    online, free or even partially free and is something awesome, yes I admit
    I would love to leech it more. PoorMan's DataStore is an expression of such
    desires. Put simply, the desire of not having to invest. But that's a lie,
    you see investment not always has to do with money, it includes time and
    energy too.


Purpose -
---------

    To allow people to use google appengine's DataStore in a distributed way,
    so as to be able to store more than a gigabyte of data - the current quota
    on free users.
    
    To provide an API so that anyone can easily configure, reuse and hack ahead.
    
    To demonstrate the above mentioned purpose with help of a live example that
    utilizes the PoorMan API.

    To save my time and not to provide any concrete security measures, or
    methods to verify or restrict access, all that is left upto you.


Definitions -
-------------

    Slave -
        This is an appengine app, whose sole purpose is to store the data given
        to it, give information about number of bytes stored in its DataStore,
        to delete/modify/serve data when asked to. It would also take care of
        the mimetypes and serve content appropriately.
        
    Master -
        This where you utilize the PoorMan API to store or retrieve data on
        Slaves. It stores metadata on what is stored where.
        This is also where your webapp resides. The PoorMan API and your app
        have a symbiotic relationship.


Models -
--------

    On Master -
        MasterDataStore
            keyname, slave, slave_key_name

        PoorSlaves
            slave, bytes

    * NOTE: slave always representa hostname eg. foo.appspot.com and not url


    On Slave -
        DataStore
            data, mime


API -
-----

    On Master -
    
        >   set_data(_key, data, mime):                                     [+]t
                if _key exists in MasterDataStore:
                    call slave_url/set/ pass keyname=_key
                else
                    from slaves, select best one
                    call slave_url/set/ pass keyname='None'

                on success make [ keyname, slave_url, slave_key_name ] entry
                on fail notify

        >   get_url(_key):                                                  [+]t
                look for which slave has _key
                return http://slave/get/slave_key_name

        >   get_data(_key):                                                 [+]t
                url = get_url(_key)
                fetch(url), return data
                
            * NOTE - this is very slow and resource consuming and not advisable
                    instead try using ajax or get_url intelligently

        >   del_data(_key):                                                 [+]t
                if _key exists:
                    fetch ( http://slave_url/del/slave_key_name )

        >   update_stats():                                                 [+]t
                for each slave:
                    query for free bytes
                return a JSON

        >   get_upload_box(slave,extra):                                    [+]t
                returns a basic form with a file upload element with appropriate
                target, methods, etc.
                target - http://slave_url/upload/
                extra - can be used to put class, style, etc into form

        /add_upload                                                         [+]
            verify link
            add meta info
            

    On Slave -

        * NOTE - some calls here would look for Referer in http headers.
                 This is a poor form of security, but as for now this is
                 what is there.

        /set                                                                [+]t
            verify master
            determine mime
            if keyname='None'
                add [data,mime] to own datastore
            else
                verify link
                replace [data,mime] appropriately
            return the keyname

        /get/slave_key_name                                                 [+]t
            print out data with appropriate mime
            * free for all

        /del/salve_key_name                                                 [+]t
            verify link
            delete key

        /update                                                             [+]t
            verify link
            return number of used bytes

        /upload                                                             [+]t
            verify master
            add [data,mime]
            redirect to http://master/add_upload
                pass link, added key


Legend -
--------

    [ ]     - unfinished
    [+]     - implemented
    [+]t    - tested