/
README
156 lines (113 loc) · 5.24 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
Thu Dec 24 07:16:53 IST 2009 Abhishek Mishra <ideamonk at gmail.com>
-=========================-
| PoorMan's DataStore |
-=========================-
Preface/Bullshit -
------------------
Yes I'm a Freetard, and my crime is of having a leecher's mentality. Maybe
I don't have the same attitude towards a penny in my hand, or anything that
has to do with the real world. But when it comes to something that is
online, free or even partially free and is something awesome, yes I admit
I would love to leech it more. PoorMan's DataStore is an expression of such
desires. Put simply, the desire of not having to invest. But that's a lie,
you see investment not always has to do with money, it includes time and
energy too.
Purpose -
---------
To allow people to use google appengine's DataStore in a distributed way,
so as to be able to store more than a gigabyte of data - the current quota
on free users.
To provide an API so that anyone can easily configure, reuse and hack ahead.
To demonstrate the above mentioned purpose with help of a live example that
utilizes the PoorMan API.
To save my time and not to provide any concrete security measures, or
methods to verify or restrict access, all that is left upto you.
Definitions -
-------------
Slave -
This is an appengine app, whose sole purpose is to store the data given
to it, give information about number of bytes stored in its DataStore,
to delete/modify/serve data when asked to. It would also take care of
the mimetypes and serve content appropriately.
Master -
This where you utilize the PoorMan API to store or retrieve data on
Slaves. It stores metadata on what is stored where.
This is also where your webapp resides. The PoorMan API and your app
have a symbiotic relationship.
Models -
--------
On Master -
MasterDataStore
keyname, slave, slave_key_name
PoorSlaves
slave, bytes
* NOTE: slave always representa hostname eg. foo.appspot.com and not url
On Slave -
DataStore
data, mime
API -
-----
On Master -
> set_data(_key, data, mime): [+]t
if _key exists in MasterDataStore:
call slave_url/set/ pass keyname=_key
else
from slaves, select best one
call slave_url/set/ pass keyname='None'
on success make [ keyname, slave_url, slave_key_name ] entry
on fail notify
> get_url(_key): [+]t
look for which slave has _key
return http://slave/get/slave_key_name
> get_data(_key): [+]t
url = get_url(_key)
fetch(url), return data
* NOTE - this is very slow and resource consuming and not advisable
instead try using ajax or get_url intelligently
> del_data(_key): [+]t
if _key exists:
fetch ( http://slave_url/del/slave_key_name )
> update_stats(): [+]t
for each slave:
query for free bytes
return a JSON
> get_upload_box(slave,extra): [+]t
returns a basic form with a file upload element with appropriate
target, methods, etc.
target - http://slave_url/upload/
extra - can be used to put class, style, etc into form
/add_upload [+]
verify link
add meta info
On Slave -
* NOTE - some calls here would look for Referer in http headers.
This is a poor form of security, but as for now this is
what is there.
/set [+]t
verify master
determine mime
if keyname='None'
add [data,mime] to own datastore
else
verify link
replace [data,mime] appropriately
return the keyname
/get/slave_key_name [+]t
print out data with appropriate mime
* free for all
/del/salve_key_name [+]t
verify link
delete key
/update [+]t
verify link
return number of used bytes
/upload [+]t
verify master
add [data,mime]
redirect to http://master/add_upload
pass link, added key
Legend -
--------
[ ] - unfinished
[+] - implemented
[+]t - tested