forked from basho/riak_search
/
TRANSITION
317 lines (242 loc) · 10.7 KB
/
TRANSITION
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
Migrating to Riak 0.11 from earlier versions of Riak
====================================================
Table of Contents
=================
1 Riak 0.10 to 0.11
1.1 Customized configuration, not switching backends
1.2 Default configuration, or switching backends
1.2.1 Backup/Restore
1.2.2 Online Rolling Upgrade
2 Riak 0.9 and earlier to 0.11
3 Migrating from Riak 0.9.x to 0.10
4 Overview
5 Requirements
5.1 Erlang/OTP R13B04
5.2 Mercurial
6 Dependencies
6.1 Mochiweb, Webmachine, Erlang_js
6.2 make deps
7 Source
7.1 Core/KV Split
7.2 Clients
8 Configuration
8.1 app.config
8.2 Ring Storage
8.3 Your Data
1 Riak 0.10 to 0.11
~~~~~~~~~~~~~~~~~~~~
1.1 Customized configuration, not switching backends
=====================================================
If you have customized your configuration file (etc/app.config),
and you will be keeping your customized configuration file, your
transition should be as easy as stopping the 0.10 node, installing
the 0.11 binaries, and starting the node back up. The node should
find its ring and data in the same location as before, and will
continue working as if nothing changed.
1.2 Default configuration, or switching backends
=================================================
If you're using the default Riak configuration file, you'll need to
be aware that the default storage backend has changed from DETS to
Basho's new Bitcask. You'll need to choose one of two upgrade
paths in order to migrate your data from your old storage backend
to the new one (these steps will also work, in general, from any
backend to any other backend). Your options are: backup/restore,
and online rolling upgrade.
1.2.1 Backup/Restore
---------------------
The easiest way to migrate from one storage backend to another is
to backup the cluster as it stands, shut it down, switch the
backend (by upgrading or editing the config file), start the
cluster back up, and then restore the data to the cluster. Follow
these steps to do this:
1. With a running cluster, use the "riak-admin backup" command to
backup the cluster's data:
$ riak-admin backup riak@127.0.0.1 riak riak-backup.dat all
This script will exit when backup is complete. If your cluster
has a lot of data, you can backup each node individually by
switching "all" to "node" in the command line above, and then
running the command once for each node, changing the node name
in the command line.
2. Shut down the cluster.
$ riak stop
Run this command on each node.
3. Upgrade each node by installing the new 0.11 package. If
you're just switching backends (not upgrading), modify the
configuration file in this step.
4. Start the cluster
$ riak start
Run this command on each node. The nodes will automatically
reconnect to each other.
5. Reload the cluster's data
$ riak-admin restore riak@127.0.0.1 riak riak-backup.dat
If you used the whole-cluster ("all") backup mode, you're now
finished. If you backed up each node individually, you'll need
to run this command for each of those backup files (note that
they have node names appended to what you provided on the
backup command line).
Once the restore script exits, all data should be restored to the
cluster. After checking that this is true, you may remove the
data directories for the old storage backend.
1.2.2 Online Rolling Upgrade
-----------------------------
If you need your Riak cluster to remain running during the backend
transition, and you have spare network capacity, you can use
Riak's easy node addition/removal process to switch backends.
1. Configure a new node, with the new backend you want to use, and
add it to the cluster.
newnode$ riak start
newnode$ riak-admin join oldclusternode@10.0.0.100
2. Remove one of the old-backend nodes from the cluster.
oldnode0$ riak-admin leave
3. When the old-backend node completes handoff, it will exit (use
'ps' or similar to watch for it to shut down). Once it has
exited, upgrade to 0.11 (or switch the backend) and restart the
node.
4. Repeat steps 2 and 3 for each other old-backend node in the
cluster.
5. When all nodes are running on the new backend, shutdown the new
node you set up in step 1 (if you wish, or leave it up if you like).
Note: You can skip step 1 if you are running a cluster of more
than one node, and your cluster can tolerate (capacity- and
throughput-wise) being temporarily one node smaller. Step 1 is
merely an attempt to add extra capacity to the system before
beging an expensive operation.
2 Riak 0.9 and earlier to 0.11
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you are upgrading from a Riak version earlier than 0.10, your
only option for upgrade is backup and restore. Please use the
backup/restore method described in the 0.10-to-0.11-transition
section above.
You will also be interested in the following section, which
discusses migration from 0.9 to 0.10.
3 Migrating from Riak 0.9.x to 0.10
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 Overview
~~~~~~~~~~~
Riak has undergone significant restructuring in the transition from
version 0.9.x to version 0.10. If you are using the binary builds,
please skip directly to the Configuration and Clients sections. If
you are building from source yourself, please review the whole of
this document.
NOTE: If the only files you have changed in your Riak source clone
are those underneath the "rel" directory
("rel/overlay/etc/app.config", for example), the safest way to
update is to make a fresh clone of the Riak repository, and then
skip to the Configuration and Clients sections of this document for
details about migrating your configurations and data.
5 Requirements
~~~~~~~~~~~~~~~
5.1 Erlang/OTP R13B04
======================
Riak 0.10 uses new features ("NIFs") provided by the latest
Erlang/OTP release, R13B04. If you are building from source, you
will need this release or a newer one.
5.2 Mercurial
==============
Riak 0.10 has moved several of its components into external
repositories. If you are building from source, you will need
Mercurial installed to allow the Rebar build system to retrieve
code from these external repositories.
6 Dependencies
~~~~~~~~~~~~~~~
6.1 Mochiweb, Webmachine, Erlang_js
====================================
mochiweb, webmachine, and erlang_js are now pulled into the "deps"
subdirectory, instead of being included in the "apps" subdirectory.
If you are pulling 0.10 code into a repository that formerly had
0.9.x in it, please remove the apps/mochiweb, apps/webmachine, and
apps/erlang_js directories from your source tree.
There is a chance that your update will also leave an "apps/riak"
directory hanging around. If it does, please remove this directory
(Riak code has moved into the "apps/riak_core" and "apps/riak_kv"
directories).
6.2 make deps
==============
The "all" make target (and, by extension, the "rel" target as
well), depend on a new "deps" target, which handles the fetching
the dependencies (mochiweb, webmachine, erlang_js).
7 Source
~~~~~~~~~
7.1 Core/KV Split
==================
We've drawn a line through 0.9.x Riak, and divided it into two
things, one called "riak_core", and the other called "riak_kv".
The things that live in riak_core are those that deal with cluster
membership. Ring-claiming and the like.
The things that live in riak_kv are those that deal with storing
data. Get and Put FSMs, backends, etc.
7.2 Clients
============
We've also moved the clients out of the client_lib subdirectory,
and into their own language-specific repositories on BitBucket. At
[http://bitbucket.org/basho/], you should find:
+ riak-python-client
+ riak-php-client
+ riak-erlang-client
+ riak-java-client
+ riak-javascript-client
+ riak-ruby-client
8 Configuration
~~~~~~~~~~~~~~~~
8.1 app.config
===============
Splitting the "riak" Erlang application into the "riak_core" and
"riak_kv" Erlang applications means that configuration options for
each component need to move around in etc/app.config.
Where before etc/app.config would have contained a section like:
{riak, [
%% many settings here
]},
Now, etc/app.config should contain two sections like:
{riak_core, [
%% core-specific settings
]},
{riak_kv, [
%% kv-specific settings
]},
The list of settings that moved to the riak_core section are:
+ choose_claim_fun
+ cluster_name - string, defaults to "default"
+ default_bucket_props
+ gossip_interval - integer, defaults to 60k msec
+ ring_creation_size - integer, defaults to 64
+ ring_state_dir - string
+ target_n_val - integer, defaults to 3
+ wants_claim_fun
+ web_ip - string. Used to be "riak_web_ip"
+ web_logdir - string.
+ web_port - integer. Used to be "riak_web_port"
IMPORTANT: Note the rename of "riak_web_*" to just "web_*"
The list of settings that moved to the riak_kv section are:
+ add_paths - list, defaults to []
+ handoff_concurrency - integer, defaults to 4
+ js_source_dir - string
+ js_vm_count - integer
+ mapred_name - string
+ raw_name - string
+ riak_kv_stat - boolean.
+ stats_urlpath - string
+ storage_backend - atom. Backend names are now prefixed as "riak_kv_" instead of just "riak_".
+ pb_ip - string
+ pb_port - integer
IMPORTANT: The default backend has changed names from
riak_dets_backend to riak_kv_dets_bakend. Other backends have
changed names as well. This rename does not affect you if you are
using the Innostore backend.
If you did not have any of these settings defined in etc/app.config,
you still do not need to define them in your new etc/app.config.
8.2 Ring Storage
=================
Periodically, Riak nodes save the state of their ring to disk. In
0.9, these files were named "data/ring/riak_ring.*", but in 0.10,
they're named "data/ring/riak_core_ring.*". Renaming the old files
to the new scheme is all you need to do to make the switch.
If you referenced any Riak modules in your bucket properties, you
will also need to change those references to point to the new
module names after your cluster is running.
8.3 Your Data
==============
The rest of your cluster's data, stored in the "data" directory
("data/dets" or "data/innodb", for example)
should be safe to either leave in place, or copy to your new
install location, depending on how you upgraded.