Add quick tutorial

CMakeLists.txt

@@ -59,6 +59,7 @@ endif ()
 # === Includes ===
 include_directories(BEFORE ./)
 include_directories(BEFORE ${ASIO_INCLUDE_DIR})
+include_directories(BEFORE ${PROJECT_SOURCE_DIR}/include)
 include_directories(BEFORE ${PROJECT_SOURCE_DIR}/include/libnuraft)
 include_directories(BEFORE ${PROJECT_SOURCE_DIR}/examples)
 include_directories(BEFORE ${PROJECT_SOURCE_DIR}/examples/calculator)

docs/async_replication.md

@@ -5,7 +5,7 @@ The original Raft's commit happens after reaching quorum, which means that netwo
 
 However, there are some loosened cases that we want to achieve low latency by sacrificing consistency and resolving conflicts manually. Then waiting for the acknowledges from a majority of servers is a waste of time.
 
-To support such cases, we provide `async_replication_` flag in [`cluster_config`](../include/cluster_config.hxx). If that flag is set, `append_entries()` API immediately returns with the result of `state_machine::pre_commit()`, and replication is done in background later.
+To support such cases, we provide `async_replication_` flag in [`cluster_config`](../include/libnuraft/cluster_config.hxx). If that flag is set, `append_entries()` API immediately returns with the result of `state_machine::pre_commit()`, and replication is done in background later.
 
 Below diagram shows the overall flow. You can compare it with [original sequence](replication_flow.md):
 ```
@@ -29,7 +29,7 @@ X------>|       |   raft_server::append_entries()
 
 To enable asynchronous replication, `state_machine::pre_commit()` function should do the actual execution, instead of `state_machine::commit()`. In addition to that, you also should implement `state_machine::rollback()` correctly, to revert any changes done by `pre_commit()`.
 
-In synchronous replication mode, we provide another option: `async_handler` in [`raft_params`](../include/raft_params.hxx). Here are the differences between asynchronous replication mode and synchronous replication with `async_handler` mode:
+In synchronous replication mode, we provide another option: `async_handler` in [`raft_params`](../include/libnuraft/raft_params.hxx). Here are the differences between asynchronous replication mode and synchronous replication with `async_handler` mode:
 
 * Synchronous replication with `async_handler` mode:
     * The actual execution in state machine happens after reaching consensus.

docs/basic_operations.md

@@ -5,21 +5,21 @@ Basic Operations
 
 Initializing Raft Server
 ------------------------
-First of all, you need to define your own [log store](../include/log_store.hxx), [state machine](../include/state_machine.hxx), and [state manager](../include/state_mgr.hxx) (optionally [debugging logger](../include/logger.hxx)):
+First of all, you need to define your own [log store](../include/libnuraft/log_store.hxx), [state machine](../include/libnuraft/state_machine.hxx), and [state manager](../include/libnuraft/state_mgr.hxx) (optionally [debugging logger](../include/libnuraft/logger.hxx)):
 ```C++
 ptr<logger> my_logger;
 ptr<state_machine> my_state_machine;
 ptr<state_mgr> my_state_manager;
 ```
 Log store will not be passed at initialization time, but will be loaded through `load_log_store()` API in state manager later. So you need properly implement that function.
 
-After that, set your [Asio](../include/asio_service_options.hxx) and [Raft](../include/raft_params.hxx) options:
+After that, set your [Asio](../include/libnuraft/asio_service_options.hxx) and [Raft](../include/libnuraft/raft_params.hxx) options:
 ```C++
 asio_service::options asio_opt;
 raft_params params;
 ```
 
-And then you can use [Launcher](../include/launcher.hxx) for initialization:
+And then you can use [Launcher](../include/libnuraft/launcher.hxx) for initialization:
 ```C++
 ptr<raft_server> server = launcher.init(my_state_machine,
                                         my_state_manager,
@@ -45,7 +45,7 @@ Once you initialize Raft server, it will invoke below APIs from your custom modu
     * This function should return the last committed Raft cluster config, that contains the membership info.
     * At the very first launch, you can return a cluster config that contains the server itself only. After adding server the cluster config will change, and you should make it durable (if necessary).
 * `state_mgr::read_state()`
-    * This function should return the last [server state](../include/srv_state.hxx), that contains term and voting info.
+    * This function should return the last [server state](../include/libnuraft/srv_state.hxx), that contains term and voting info.
 * `state_machine::last_commit_index()`
     * You should make the last committed log number durable (if necessary), and return it here. Otherwise, Raft server attempts to do catch-up from the beginning.
 * `state_machine::last_snapshot()`
@@ -54,15 +54,15 @@ Once you initialize Raft server, it will invoke below APIs from your custom modu
 
 Shutting Down Raft Server
 -------------------------
-You can simply use [Launcher](../include/launcher.hxx)'s shutdown API:
+You can simply use [Launcher](../include/libnuraft/launcher.hxx)'s shutdown API:
 ```C++
 bool success = launcher.shutdown();
 ```
 This API is a blocking call, so that the server termination is guaranteed once the function returns `true`.
 
 Adding Server -- Organizing a Group
 ---
-Set [`srv_config`](../include/srv_config.hxx) of the server to be added:
+Set [`srv_config`](../include/libnuraft/srv_config.hxx) of the server to be added:
 ```C++
 srv_config server_to_add(...);
 ```
@@ -89,7 +89,7 @@ The server to be removed should be running at the time you remove server. Otherw
 
 Appending Log -- Replication
 ---
-You can allocate [`buffer`](../include/buffer.hxx), and put your data into it using [`buffer_serializer`](../include/buffer_serializer.hxx):
+You can allocate [`buffer`](../include/libnuraft/buffer.hxx), and put your data into it using [`buffer_serializer`](../include/libnuraft/buffer_serializer.hxx):
 ```C++
 ptr<buffer> b = buffer::alloc(...);
 buffer_serializer s(b);

docs/custom_quorum_size.md

@@ -7,7 +7,7 @@ The motivation comes from [Flexible Paxos](https://fpaxos.github.io/) paper:
 
 The basic idea is that as long as there is at least one overlapping node between the quorum for commit and the quorum for leader election, the entire group is safe. For example, let's say Qc and Qe represent the size of quorum for commit and leader election, respectively. If we have 5 servers, the set of {Qc, Qe} pairs {1, 5}, {2, 4}, {3, 3} (original algorithm), {4, 2}, and {5, 1} provides the same level of safety. Note that availability will be sacrificed as the value of |Qc - Qe| increases.
 
-For custom quorum size, we provide two parameters: `custom_commit_quorum_size_` and `custom_election_quorum_size_` in [`raft_params`](../include/raft_params.hxx).
+For custom quorum size, we provide two parameters: `custom_commit_quorum_size_` and `custom_election_quorum_size_` in [`raft_params`](../include/libnuraft/raft_params.hxx).
 
 ```C++
 raft_params params;

docs/dealing_with_buffer.md

@@ -1,7 +1,7 @@
 Dealing with Buffer
 ---
 
-[`buffer`](../include/buffer.hxx) class instance just points to a raw memory blob. At the beginning of each memory blob, a few bytes are reserved for metadata: 4 bytes if buffer size is less than 32KB and 8 bytes otherwise. User data section starts right after that, thus you should write your data starting from there.
+[`buffer`](../include/libnuraft/buffer.hxx) class instance just points to a raw memory blob. At the beginning of each memory blob, a few bytes are reserved for metadata: 4 bytes if buffer size is less than 32KB and 8 bytes otherwise. User data section starts right after that, thus you should write your data starting from there.
 
 You can use `alloc()` API to allocate memory:
 ```C++
@@ -29,7 +29,7 @@ Buffer Serializer
 ---
 `buffer` itself has a few APIs to get and put data, but we do not recommend using those APIs, since they change the internal position of the buffer, which blocks concurrent reads and also is mistake-prone if you forget to reset the position.
 
-Instead, you can use [`buffer_serializer`](../include/buffer_serializer.hxx):
+Instead, you can use [`buffer_serializer`](../include/libnuraft/buffer_serializer.hxx):
 ```C++
 ptr<buffer> b = buffer::alloc( size_to_allocate );
 buffer_serializer s(b);

docs/enabling_ssl.md

@@ -1,7 +1,7 @@
 Enabling SSL/TLS
 ----------------
 
-When you set [`asio_options`](../include/asio_service_options.hxx), there are a few options for enabling SSL/TLS:
+When you set [`asio_options`](../include/libnuraft/asio_service_options.hxx), there are a few options for enabling SSL/TLS:
 ```C++
 asio_service::options asio_opt;
 asio_opt.enable_ssl_        = true;

docs/how_to_use.md

@@ -6,7 +6,7 @@ The fundamental logic is described in the original paper. We skip the details he
 
 * Diego Ongaro and John K. Ousterhout, [In Search of an Understandable Consensus Algorithm](https://raft.github.io/raft.pdf), USENIX ATC 2014.
 
-If you are in a hurry, go to [Quick Start Guide](quick_start_guide.md).
+If you are in a hurry, go to [Quick Start Guide](quick_start_guide.md) or [Quick Tutorial](quick_tutorial.md).
 
 
 Modules
@@ -15,17 +15,17 @@ It basically consists of 5 modules: Raft server, Asio layer, log store, state ma
 * Raft server: coordinating all incoming requests and responses from users and other nodes.
 * Asio layer: dealing with network communication and timer, as well as thread pool management.
 * Log store: managing read, write, and compact operations of Raft logs.
-    * [Interface](../include/log_store.hxx)
+    * [Interface](../include/libnuraft/log_store.hxx)
     * [Example - in-memory log store](../examples/in_memory_log_store.cxx)
 * State machine: executing commit (optionally pre-commit and rollback), and managing snapshots.
-    * [Interface](../include/state_machine.hxx)
+    * [Interface](../include/libnuraft/state_machine.hxx)
     * [Example #1 - echo state machine](../examples/echo/echo_state_machine.hxx)
     * [Example #2 - calculator state machine](../examples/calculator/calc_state_machine.hxx)
 * State manager: saving and loading cluster configuration and status.
-    * [Interface](../include/state_mgr.hxx)
+    * [Interface](../include/libnuraft/state_mgr.hxx)
     * [Example - in-memory state manager](../examples/in_memory_state_mgr.hxx)
 * (Optional) Debugging logger: for system logging.
-    * [Interface](../include/logger.hxx)
+    * [Interface](../include/libnuraft/logger.hxx)
     * [Example - example logger](../examples/logger_wrapper.hxx)
 
 

docs/leader_election_priority.md

@@ -8,7 +8,7 @@ The main goal is to prioritize member nodes for next leader election. Raft uses
 
 New Definitions
 ---------------
-We define `priority_` in [`srv_config`](../include/srv_config.hxx), which is an integer from 0 to any value. Priority `0` is a special value so that a node with zero priority will never be a leader. All nodes in the same Raft group will be aware of all members' priority.
+We define `priority_` in [`srv_config`](../include/libnuraft/srv_config.hxx), which is an integer from 0 to any value. Priority `0` is a special value so that a node with zero priority will never be a leader. All nodes in the same Raft group will be aware of all members' priority.
 
 Each node has an internal local value *target priority* which is initially set to `max(priority of all nodes)`.
 

docs/leadership_expiration.md

@@ -21,7 +21,7 @@ To address this issue, we introduce leadership expiration.
 
 How To Enable Leadership Expiration
 -----------------------------------
-We provide a parameter `leadership_expiry_` in [`raft_params`](../include/raft_params.hxx) to set the expiration time (in millisecond) of the current leader:
+We provide a parameter `leadership_expiry_` in [`raft_params`](../include/libnuraft/raft_params.hxx) to set the expiration time (in millisecond) of the current leader:
 
 ```C++
 raft_params params;

docs/quick_start_guide.md

@@ -4,10 +4,11 @@ Quick Start Guide
 [examples/quick_start.cxx](../examples/quick_start.cxx)
 
 ```C++
-#include "nuraft.hxx"
 #include "echo_state_machine.hxx"
 #include "in_memory_state_mgr.hxx"
 
+#include "libnuraft/nuraft.hxx"
+
 #include <chrono>
 #include <string>
 #include <thread>
@@ -52,4 +53,4 @@ int main(int argc, char** argv) {
 }
 ```
 
-Please refer to [Echo server](../examples/echo) example for more details.
+Please refer to [Quick Tutorial](quick_tutorial.md) and [Echo server](../examples/echo) example for more details.

docs/quick_tutorial.md

@@ -0,0 +1,61 @@
+How to Use Nuraft: A Quick Tutorial
+===
+
+If you are ready to explore NuRaft, the best starting point to learn using NuRaft is the examples within the [examples directory](../examples) in the repo. There are two examples: [echo](../examples/echo) and [calculator](../examples/calculator), both share some common files. For a hello-world equivalent example, see [Quick Start Guide](quick_start_guide.md).
+
+The major tasks to use NuRaft include the following:
+
+### Providing a Log Store ###
+
+[in_memory_log_store.hxx](../examples/in_memory_log_store.hxx) and [in_memory_log_store.cxx](../examples/in_memory_log_store.cxx) provide a log store example for both examples.
+
+### Defining the State Machine ###
+
+Defining log entry format, and state machine actions, and providing state machine snapshot support. See [echo/echo_state_machine.hxx](../examples/echo/echo_state_machine.hxx) and [calculator/calc_state_machine.hxx](../examples/calculator/calc_state_machine.hxx) for the examples.
+
+### Server state and cluster configuration ###
+
+There is a cluster-wide configuration [cluster_config](../include/libnuraft/cluster_config.hxx), which contains a list of [srv_config](../include/libnuraft/srv_config.hxx) for each server. Class [state_mgr](../include/libnuraft/state_mgr.hxx) manages the configuration and [srv_state](../include/libnuraft/srv_state.hxx). You need to override the base class with custom specifics. See [in_memory_state_mgr.hxx](../examples/in_memory_state_mgr.hxx) for example.
+
+### Determining members and their parameters ###
+
+In [example_common.hxx](../examples/example_common.hxx) there is code to define a server, and initialize raft protocol with parameters (init_raft).
+
+Assuming we have a declaration
+```C++
+raft_params params;
+```
+Some important protocol parameters to consider are the following:
+
+- `params.heart_beat_interval_`: a leader will send a heartbeat message to followers if this interval (in milliseconds) has passed since the last replication or heartbeat message.
+
+- `params.election_timeout_lower_bound_` and `params.election_timeout_upper_bound_`: they determine the time (in milliseconds, between the lower and upper) a follower will wait before initiating leader election. These three parameters together determine how long a leader failure will be detected. When a leader fails, writes will be temporarily unavailable until a new leader is elected. The average lapse will be about half way between this timeout interval.
+
+- `params.reserved_log_items_`: the number of trailing log entries will be preserved when a snapshot is taken. If one member falls behind the others temporarily, it needs to catch up from the leader using the Raft log. If the log is truncated too soon, the member would have to use a snapshot to catch up due to needed log entries being unavailable. Snapshot based recovery is costly, especially if the data volume is large. Keeping enough trailing log entries will help avoid the costly snapshot-based catch up.
+
+- `params.snapshot_distance_`: snapshot frequency (in number of log entries). When a member restarts or recovers from a remote snapshot, it will replay the log entries after the snapshot. Frequent snapshots will reduce the number of log entries to play, but incurs more overhead of snapshots. On the other hand, less frequent snapshot may increase the time of restart or catch up with less cost of snapshots.
+
+
+### Servers ###
+
+[echo/echo_server.cxx](../examples/echo/echo_server.cxx) and [calculator/calc_server.cxx](../examples/calculator/calc_server.cxx) contains both server and user interface code.
+
+### Starting or restarting a cluster and adding a member ###
+
+You start servers and then add to the cluster. [Launcher](../include/libnuraft/launcher.hxx) can help to start Raft. See `add_server()` and `server_list()` in [example_common.hxx](../examples/example_common.hxx) for example.
+
+### Removing a member and shutting down the cluster ###
+
+You remove a server, and then shut it down. Use launcher for shutdown, see server code in response to the quit command.
+
+### Client API Considerations ###
+
+The examples contain user interface code in the main server. If you use a client-server model, you need to define the communication API, say using [gRPC](https://grpc.io/) for example. You need to expose server role status to the client for it to differentiate the leader from the followers.
+
+- Writes will go to the leader only. Raft can be configured to forward requests to the leader, but this may incur more hops, thus longer latency. If a node is no longer a leader, it can return the current leader to the client.
+
+- Reads from the leader for latest values.
+
+- Reads from the followers for the values that may be delayed.
+
+For more details and advanced topics, please refer to the [How to Use](../docs/how_to_use.md) document.

docs/readonly_member.md

@@ -5,7 +5,7 @@ Read-only member is a node who does not initiate or participate in leader electi
 
 Read-only member is useful when you want to replicate data to geo-distributed nodes in remote datacenter. If you set those nodes as read-only members, they are not counted in quorum so that you can still organize a quorum within the same datacenter, which keeps commit latency relatively low. Remote nodes will catch-up the leader eventually.
 
-To make a member read-only, you need to set `learner_` flag to `true`, when you generate [`srv_config`](../include/srv_config.hxx):
+To make a member read-only, you need to set `learner_` flag to `true`, when you generate [`srv_config`](../include/libnuraft/srv_config.hxx):
 ```C++
 srv_config normal_member(1, 0, "10.10.10.1:12345", "", false);
 srv_config       learner(2, 0, "10.10.10.2:12345", "", true);

docs/snapshot_transmission.md

@@ -43,8 +43,8 @@ X---------->|   send {Y, D_Y}
 |           |
 ```
 
-Once a new empty node joins the Raft group, or there is a lagging node whose the last log is older than the last log compaction, Raft starts to send a snapshot. It first calls [`read_logical_snp_obj()`](../include/state_machine.hxx) with `obj_id = 0` in leader's side. Then your state machine needs to return corresponding data (i.e., a binary blob) `D_0`, and `{0, D_0}` pair will be sent to the follower who receives the snapshot.
+Once a new empty node joins the Raft group, or there is a lagging node whose the last log is older than the last log compaction, Raft starts to send a snapshot. It first calls [`read_logical_snp_obj()`](../include/libnuraft/state_machine.hxx) with `obj_id = 0` in leader's side. Then your state machine needs to return corresponding data (i.e., a binary blob) `D_0`, and `{0, D_0}` pair will be sent to the follower who receives the snapshot.
 
-Once the follower receives the object, it invokes [`save_logical_snp_obj()`](../include/state_machine.hxx) with received object ID and data. Your state machine properly stores the received data, and then change the given `obj_id` value to the next object ID. That new object ID will be sent to leader, and leader will invoke `read_logical_snp_obj()` with the new ID.
+Once the follower receives the object, it invokes [`save_logical_snp_obj()`](../include/libnuraft/state_machine.hxx) with received object ID and data. Your state machine properly stores the received data, and then change the given `obj_id` value to the next object ID. That new object ID will be sent to leader, and leader will invoke `read_logical_snp_obj()` with the new ID.
 
 Above process will be repeated until `read_logical_snp_obj()` sets `is_last_obj = true`. Once `is_last_obj` is set to `true`, follower's Raft will call `save_logical_snp_obj()` with the given data, and then call `apply_snapshot()` where your implementation needs to replace the data in state machine with the newly received one.