Skip to content

Latest commit

 

History

History
194 lines (153 loc) · 9.76 KB

ha.adoc

File metadata and controls

194 lines (153 loc) · 9.76 KB

High Availability

edit

ArcadeDB supports a High Availability mode where multiple servers share the same database (replication).

In order to start an ArcadeDB server with the support for replication, you need to:

  1. Enable the HA module by setting the configuration arcadedb.ha.enabled to true

  2. Define the servers that are part of the clusters (if you are using Kubernetes, look at Kubernetes paragraph). To setup the server list, define the arcadedb.ha.serverList setting by separating the servers with commas (,) and using the following format for servers: <hostname/ip-address[:port]>. The default port is 2424 if not specified.

  3. Define the local server name by setting the arcadedb.server.name configuration. Each node must have a unique name. If not specified, the default server name is "ArcadeDB_0"

Example:

$ bin/server.sh -Darcadedb.ha.enabled=true
                -Darcadedb.server.name=FloridaServer
                -Darcadedb.ha.serverList=192.168.0.2,192.168.0.1:2424,japan-server:8888

The log should look like this:

[HttpServer] <FloridaServer> - HTTP Server started (host=0.0.0.0 port=2480)
[LeaderNetworkListener] <ArcadeDB_0> Listening for replication connections on 127.0.0.1:2424 (protocol v.-1)
[HAServer] <FloridaServer> Unable to find any Leader, start election (cluster=arcadedb configuredServers=1 majorityOfVotes=1)
[HAServer] <FloridaServer> Change election status from DONE to VOTING_FOR_ME
[HAServer] <FloridaServer> Starting election of local server asking for votes from [] (turn=1 retry=0 lastReplicationMessage=-1 configuredServers=1 majorityOfVotes=1)
[HAServer] <FloridaServer> Current server elected as new Leader (turn=1 totalVotes=1 majority=1)
[HAServer] <FloridaServer> Change election status from VOTING_FOR_ME to LEADER_WAITING_FOR_QUORUM
[HAServer] <FloridaServer> Contacting all the servers for the new leadership (turn=1)...

At startup, the ArcadeDB server will look for an existent cluster to join based on the configured list of servers, otherwise a new cluster will be created. In this example we set the server name to FloridaServer.

Every time a server joins a cluster, it starts the process to elect the new leader. If the cluster exists and already contains a Leader, then the existent leader is kept. Every time a server leaves the cluster (because it becomes unreachable), the election process is started again. To know more about the RAFT election process, look at RAFT protocol.

Note
The cluster name by default is "arcadedb", but you can have multiple clusters in the same network. To specify a custom name, set the configuration arcadedb.ha.clusterName=<name>. Example: bin/server.sh -Darcadedb.ha.clusterName=projectB
Note
All servers of a cluster serve the same databases.

Architecture

ArcadeDB has a Leader/Replica model by using RAFT consensus for election and replication.

Cluster of Servers
 +------------+        +------------+        +------------+
 | ArcadeDB_1 |<-------| ArcadeDB_0 |------->| ArcadeDB_2 |
 |  Replica   |        |   Leader   |        |  Replica   |
 +------------+        +------------+        +------------+
       |                     |                     |
       |                     |                     |
       V                     V                     V
  +----------+          +----------+          +----------+
  |  Journal |          |  Journal |          |  Journal |
  |      {d} |          |      {d} |          |      {d} |
  +----------+          +----------+          +----------+

Each server has its own Journal. The Journal is used in case of recovery of the cluster to get the most updated replica and to align the other nodes. All the write operations must be coordinated by the Leader node.

Any read operation, like a query, can be executed by any server in the cluster, while write operations can be executed only by the Leader server.

Read Request executed on a Replica
            +------------+
            |  Client A  |
            +-----+------+
               ^  |
(2) send result|  | (1) read request
 to the client |  V
            +------------+
            | ArcadeDB_1 |
            |  Replica   |
            +------------+

ArcadeDB doesn’t mandate the clients to be connected directly to the leader to execute write operations, but will use the Replica server to forward the write request to the Leader server. Everything is transparent for the end user where both Leader and Replica servers can read and write, but internally only the read requests are executed on the connected server. All the write requests will be transparently forwarded to the Leader.

Look at the picture below where the client Client A is connected to the replica server ArcadeDB_1 where it executes a write request.

Write Request executed on a Replica
            +------------+
            |  Client A  |
            +-----+------+
               ^  |
(6) send result|  | (1) write request
 to the client |  V
            +------------+              +------------+
            | ArcadeDB_1 |------------->| ArcadeDB_0 | (3) execute write request
            |  Replica   | (2) forward  |   Leader   | (4) replicate to all the servers
            +------------+              +------------+  (including ArcadeDB_1)
                  ^                           |
                  |                           |
                  +---------------------------+
                   (5) Send result back to the
                      requesting server

Auto fail-over

ArcadeDB cluster uses a quorum to assure the integrity of the database is maintained across all the servers forming the cluster. The quorum is set by default to MAJORITY, that means the majority of the servers in the cluster must return the same result to be considered accepted and propagated to all the servers.

Note
The quorum is MAJORITY by default. You can specify a different quorum by setting the number of servers or none to have no quorum and all to wait the response from all the servers. Set the configuration arcadedb.ha.quorum=<quorum>. Example: bin/server.sh -Darcadedb.ha.quorum=all

If the configured quorum is not met, the transaction is rollback on all the servers, the database returns to the previous state and a transaction error is thrown to the client.

ArcadeDB manages the fail-over automatically in most of the cases.

Server unreachable

A server can become unreachable for many reasons:

  • The ArcadeDB Server process has been terminated

  • The physical or virtual server hosting the ArcadeDB Server process has been shut off or is rebooting

  • The physical or virtual server hosting the ArcadeDB Server process has network issues and can’t reach one or more of the other servers

  • Network issues that prevent the ArcadeDB Server to communicate with the rest of the servers in the cluster

Auto balancing clients

More coming soon.

Troubleshooting

Performance: insertion is slow

ArcadeDB uses an optimistic lock approach: if two threads try to update the same page, the first thread wins, the second thread throws a ConcurrentModificationException and forces the client to retry the transaction or fail after a certain number of retries (configurable). Often this fail/retry mechanism is totally hidden to the developer that executes a transaction via HTTP or via the Java API:

db.transaction( ()-> {
  // MY TRANSACTION CODE
});

If you are inserting a lot of record in parallel (by using the Server, or just via API multi-thread), you could benefit by allocating the bucket per thread. Example to change the bucket selection strategy for the vertex type "User" via SQL:

ALTER TYPE User BucketSelectionStrategy `thread`

With the command above, in insertion ArcadeDB will select the physical bucket based on the thread the request is coming from. If you have enough buckets (created by default when you create a new type, but you can manually adjust it) insertions can go truly in parallel with zero contentions in pages, meaning zero exception and retries.

HA Settings

The following settings are used by the High Availability module:

Setting Description Default Value

arcadedb.ha.clusterName

Cluster name. Useful in case of multiple clusters in the same network

arcadedb

arcadedb.ha.serverList

Servers in the cluster as a list of <hostname/ip-address:port> items separated by comma. Example: 192.168.0.1:2424,192.168.0.2:2424. If not specified, auto-discovery is enabled

NOT DEFINED (auto discovery is enabled by default)

arcadedb.ha.serverRole

Enforces a role in a cluster, either "any" or "replica"

"any"

arcadedb.ha.quorum

Default quorum between 'none', 1, 2, 3, 'majority' and 'all' servers

MAJORITY

arcadedb.ha.quorumTimeout

Timeout waiting for the quorum

10000

arcadedb.ha.k8s

The server is running inside Kubernetes

false

arcadedb.ha.k8sSuffix

When running inside Kubernetes use this suffix to reach the other servers. Example: arcadedb.default.svc.cluster.local

arcadedb.ha.replicationQueueSize

Queue size for replicating messages between servers

512

arcadedb.ha.replicationFileMaxSize

Maximum file size for replicating messages between servers"

1GB

arcadedb.ha.replicationChunkMaxSize

Maximum channel chunk size for replicating messages between servers

16777216

arcadedb.ha.replicationIncomingHost

TCP/IP host name used for incoming replication connections

localhost

arcadedb.ha.replicationIncomingPorts

TCP/IP port number used for incoming replication connections

2424-2433