Skip to content
Newer
Older
100644 181 lines (124 sloc) 6.09 KB
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
1
2 Ringo - Distributed key/value storage for immutable data
3 --------------------------------------------------------
4
5 Ringo is an experimental, distributed, replicating key-value store based
6 on consistent hashing and immutable data. Unlike many general-purpose
7 databases, Ringo is designed for a specific use case: For archiving
8 small (less than 4KB) or medium-size data items (<100MB) in real-time
9 so that the data can survive K - 1 disk breaks, where K is the desired
10 number of replicas, without any downtime, in a manner that scales to
11 terabytes of data. In addition to storing, Ringo should be able to
12 retrieve individual or small sets of data items with low latencies
13 (<10ms) and provide a convenient on-disk format for bulk data access.
14
15 WARNING: Ringo should not be used yet as a primary storage for critical
16 data. Due to the fact that Ringo treats all data immutable, data
17 corruption or loss should be improbable. However, Ringo needs more
18 testing in real-world settings before we can be reasonably sure that it
19 works as advertised. To make this happen, feel free to try it out, adapt
20 it to your needs, and report your experiences!
21
bc70369 Additions to README
Ville Tuulos authored
22 If you find Ringo interesting, you might also want to check out a paper
23 about Amazon's Dynamo
24
25 http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon-dynamo-sosp2007.pdf
26
27 and another Dynamo-like system, Scalaris, which is also implemented in
28 Erlang: http://code.google.com/p/scalaris/
29
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
30
31 Requirements
32 ''''''''''''
33
bc70369 Additions to README
Ville Tuulos authored
34 Backend system (required):
35
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
36 - Erlang R12B or newer
bc70369 Additions to README
Ville Tuulos authored
37 - C toolchain and autotools (for building bfile)
38 - Lighttpd or other web server that supports SCGI
39
40 Test harness / Python interface (optional):
41
42 - pkill command (Debian package procps)
43 - Python
44 - Python module pycurl (Debian package python-pycurl)
45 - Python module cjson (Debian package python-cjson)
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
46
47
48 Directories
49 '''''''''''
50
51 doc/ Documentation
52
53 ring/ Ringo backend
54 ring/bfile High-performance replacement for the Erlang's standard file
55 module
56 ring/src Backend sources
57 ring/test Test harness
58
59 ringogw/ Ringo web frontend
60 ringogw/src Frontend sources
61 ringogw/web Web interface
62 ringogw/py Python interfaces for the Ringo frontend and for the Disco
63 map/reduce framework
64
65
66 Compiling
67 '''''''''
68
69 cd ringo
70 ./compile.sh
71
72
bc70369 Additions to README
Ville Tuulos authored
73 Starting Ringo
74 ''''''''''''''
75
76 First you need to initialize a number of virtual nodes - at least one is
77 required. A virtual node is defined by an empty directory whose name is
78 a random 128-bit integer in hexadecimal notation.
79
80 A convenience script, create_node.sh, is provided that can be used to
81 setup a virtual node. For instance,
82
83 ringo/create_node.sh trurl /data/ringo
84
85 sets up a new virtual node on the host trurl to the directory
86 /data/ringo. The script uses ssh to log in to the host. It asks for a
87 password unless key-based ssh authentication is properly set up (which
88 is recommended). You may run this script on different nodes as many
89 times as you like.
90
91 You need to list all hostnames that may possibly host a Ringo node
92 in a file called ~/.hosts.erlang. For further information about this
93 file, see an Erlang manual page at "man 3erl net_adm". For instance the
94 following command creates the required file for the virtual node that
95 was initialized above:
96
97 echo "'trurl'." > ~/.hosts.erlang
98
99 After a number of virtual nodes have been initialized, Ringo may be
100 started. Again, a simple script is provided that starts up all virtual
101 nodes on a specified host. For instance,
102
103 ringo/start_nodes.sh trurl /data/ringo
104
105 starts up all nodes on the host trurl. After a while, the ring should
106 be up and running.
107
108 The web frontend provides a convenient way to monitor status of the
109 system. An example configuration file for the Lighttpd web server is
110 provided at ringo/ringogw/lighttpd.conf that communicates with the
111 frontend process over SCGI. The following script starts up the
112 web server and the frontend process:
113
114 ringo/ringogw/start_ringogw.sh
115
116 Now you should see the status page at http://localhost:15000. On the
117 status page, you can click nodes on the leftmost panel to see domains
02791aa Added a note that you need to wait for a while before you see new nod…
Ville Tuulos authored
118 that they contain. You can click a domain on the middle panel to see its
119 replicas. By clicking a replica, you can see its status. You might need
120 to wait for 10 seconds or so, and reload the page, to see new nodes and
121 domains appear.
bc70369 Additions to README
Ville Tuulos authored
122
123
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
124 Usage
125 '''''
126
bc70369 Additions to README
Ville Tuulos authored
127 You can create domain, put keys and get keys from Ringo using simple
128 HTTP requests. Assuming that you have started a ring as instructed
129 above, you can create a new domain called "foobar" with the following
130 POST request. Here curl is used to make a request but any other HTTP
131 client would work as well:
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
132
bc70369 Additions to README
Ville Tuulos authored
133 curl -d "" http://localhost:15000/mon/data/foobar?create
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
134
bc70369 Additions to README
Ville Tuulos authored
135 You can put a new key-value pair to the domain with following POST
136 request:
137
138 curl -d "testvalue" http://localhost:15000/mon/data/foobar/testkey
139
140 and retrieve the value given the key with a GET request as follows:
141
142 curl http://localhost:15000/mon/data/foobar/testkey
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
143
bc70369 Additions to README
Ville Tuulos authored
144 this returns all values assigned with the key "testkey". If only one
145 value is required, the parameter ?single can be used:
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
146
bc70369 Additions to README
Ville Tuulos authored
147 curl http://localhost:15000/mon/data/foobar/testkey?single
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
148
bc70369 Additions to README
Ville Tuulos authored
149 A Python class Ringo is provided at ringo/ringogw/py/ringogw.py that
150 encapsulates the above HTTP requests in Python function calls.
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
151
bc70369 Additions to README
Ville Tuulos authored
152 An experimental interface for Disco, an an open-source implementation
153 of the Map/Reduce framework (http://discoproject.org), can be found at
154 ringo/ringogw/py/ringodisco.py. This function, which implements the
155 Disco's map reader interface, makes it possible to use data stored in
156 Ringo as input to a Disco job. The function provides a particularly
157 efficient way of accessing data directly from Ringo's live data files.
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
158
159
bc70369 Additions to README
Ville Tuulos authored
160 Running the Test Harness
161 ''''''''''''''''''''''''
162
163 Ringo comes with a set of tests that cover all main features of the
164 system. You need to start the ringogw web frontend process to run
165 the tests, as shown above. After the frontend has started, you can
166 run the tests with the following command:
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
167
bc70369 Additions to README
Ville Tuulos authored
168 cd ringo/ring/; ./test-system.sh
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
169
170
bc70369 Additions to README
Ville Tuulos authored
171 Contact information
172 '''''''''''''''''''
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
173
bc70369 Additions to README
Ville Tuulos authored
174 Bug reports, patches, comments etc. are welcome! Contact person is
175 Ville Tuulos who can be reached at
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
176
bc70369 Additions to README
Ville Tuulos authored
177 ville.h.tuulos -a- nokia.com
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
178
bc70369 Additions to README
Ville Tuulos authored
179 or on the IRC channel #discoproject at Freenode.
5b1d49f Added README. Minor changes to management scripts.
Ville Tuulos authored
180
Something went wrong with that request. Please try again.