Add more descriptions to add a replica

tutorial/1.0.3/add-replica/index.md

@@ -54,33 +54,32 @@ First, prepare a new computer, install required softwares and configure them.
     # gem install droonga-engine
     # npm install -g droonga-http-server
     # mkdir ~/droonga
-    # cd ~/droonga
 
 Then, remember the command line you executed to generate `catalog.json` for your cluster.
 It was:
 
     (on 192.168.0.10 or 192.168.0.11)
     # droonga-engine-catalog-generate --hosts=192.168.0.10,192.168.0.11 \
-                                      --output=./catalog.json
+                                      --output=~/droonga/catalog.json
 
 For the new node, you have to generate a `custom.json` includes only one node, with same options except the `--host` option, like:
 
     (on 192.168.0.12)
     # droonga-engine-catalog-generate --hosts=192.168.0.12 \
-                                      --output=./catalog.json
+                                      --output=~/droonga/catalog.json
 
 Let's start the server.
 
     (on 192.168.0.12)
     # host=192.168.0.12
     # droonga-engine --host=$host \
                      --daemon \
-                     --pid-file=$PWD/droonga-engine.pid
+                     --pid-file=~/droonga/droonga-engine.pid
     # droonga-http-server --port=10041 \
                           --receive-host-name=$host \
                           --droonga-engine-host-name=$host \
                           --daemon \
-                          --pid-file=$PWD/droonga-http-server.pid
+                          --pid-file=~/droonga/droonga-http-server.pid
 
 Then there are two Droonga clusters on this time.
 
@@ -113,13 +112,105 @@ If you load new data via the `load` command triggered by a batch script started
 If a crawler agent adds new data via the `add` command, stop it.
 If you put a fluentd as a buffer between crawler or loader and the cluster, stop outgoing messages from the buffer. 
 
-### Duplicate data from the cluster to the new replica
+### Duplicate data from the existing cluster to the new replica
 
-TBD
+Duplicate data from the cluster alpha to the cluster beta.
+It can be done by `drndump` and `droonga-request` commands.
+(You have to install `drndump` and `droonga-client` gem packages.)
+
+    (on 192.168.0.12)
+    # drndump --host=192.168.0.10 \
+              --receiver-host=192.168.0.12 | \
+        droonga-request --host=192.168.0.12 \
+                        --receiver-host=192.168.0.12
+
+Note that you must specify the host name or the IP address of the machine via the `--receiver-host` option.
+If you run the command line on the node `192.168.0.11`, then:
+
+    (on 192.168.0.11)
+    # drndump --host=192.168.0.10 \
+              --receiver-host=192.168.0.11 | \
+        droonga-request --host=192.168.0.12 \
+                        --receiver-host=192.168.0.11
 
 ### Join the new replica to the cluster
 
-TBD
+After the duplication is successfully done, join the new replica to the existing clster.
+Re-generate the `catalog.json` on the newly joining node `192.168.0.12`, with all nodes specified via the `--hosts` option, like:
+
+    (on 192.168.0.12)
+    # droonga-engine-catalog-generate --hosts=192.168.0.10,192.168.0.11,192.168.0.12 \
+                                      --output=~/droonga/catalog.json
+
+And restart servers on the new node:
+
+    (on 192.168.0.12)
+    # kill $(cat ~/droonga/droonga-engine.pid)
+    # kill $(cat ~/droonga/droonga-http-server.pid)
+    # host=192.168.0.12
+    # droonga-engine --host=$host \
+                     --daemon \
+                     --pid-file=~/droonga/droonga-engine.pid
+    # droonga-http-server --port=10041 \
+                          --receive-host-name=$host \
+                          --droonga-engine-host-name=$host \
+                          --daemon \
+                          --pid-file=~/droonga/droonga-http-server.pid
+
+Then there are two Droonga clusters on this time.
+
+ * The existing cluster "alpha", including two replicas.
+   * `192.168.0.10`
+   * `192.168.0.11`
+ * The new cluster including three replicas.
+   Let's give a name *"charlie"* to it, for now.
+   * `192.168.0.10`
+   * `192.168.0.11`
+   * `192.168.0.12`
+
+Note that the temporary cluster named "beta" is gone.
+And, the new node `192.168.0.12` knows the cluster charlie includes three nodes, other two existing nodes don't know that.
+Because both two existing nodes think that there are only them in the cluster they belong to, any incoming request to them never delivered to the new replica `192.168.0.12` yet.
+
+Next, copy new `catalog.json` from `192.168.0.12` to others and restart servers:
+
+    (on 192.168.0.10)
+    # kill $(cat ~/droonga/droonga-engine.pid)
+    # kill $(cat ~/droonga/droonga-http-server.pid)
+    # scp 192.168.0.12:~/droonga/catalog.json ~/droonga/
+    # host=192.168.0.10
+    # droonga-engine --host=$host \
+                     --daemon \
+                     --pid-file=~/droonga/droonga-engine.pid
+    # droonga-http-server --port=10041 \
+                          --receive-host-name=$host \
+                          --droonga-engine-host-name=$host \
+                          --daemon \
+                          --pid-file=~/droonga/droonga-http-server.pid
+
+    (on 192.168.0.11)
+    # kill $(cat ~/droonga/droonga-engine.pid)
+    # kill $(cat ~/droonga/droonga-http-server.pid)
+    # scp 192.168.0.12:~/droonga/catalog.json ~/droonga/
+    # host=192.168.0.11
+    # droonga-engine --host=$host \
+                     --daemon \
+                     --pid-file=~/droonga/droonga-engine.pid
+    # droonga-http-server --port=10041 \
+                          --receive-host-name=$host \
+                          --droonga-engine-host-name=$host \
+                          --daemon \
+                          --pid-file=~/droonga/droonga-http-server.pid
+
+Then there are just one Droonga clusters on this time.
+
+ * The new cluster "charlie",including three replicas.
+   * `192.168.0.10`
+   * `192.168.0.11`
+   * `192.168.0.12`
+
+Note that the old cluster named "alpha" is gone.
+Now the new cluster "charlie" with three replicas works perfectly, instead of the old one with two replicas.
 
 ### Restart inpouring of "write" requests