Adding new node to the running cluster

[Amandine-G]

my environment running ArangoDB

I'm using the latest ArangoDB of the respective release series:

• 3.3

Mode:

• Cluster

Storage-Engine:

• rocksdb

On this operating system:

• other: centos 6.5

We now have a cluster running with 3 nodes, and we want to add another 2 new nodes to the cluster. I tried to start the node by arangodb --starter.join 65.18.27.28:8529 while the cluster is running
but failed.
the log printed were as follows:

2018/03/16 17:09:57 Starting arangodb version 0.10.4, build 553aab6
2018/03/16 17:09:57 Contacting master http://65.18.27.28:8528...
2018/03/16 17:09:57 Cannot start because of error from master: Post http://65.18.27.28:8528/hello: dial tcp 65.18.27.28:8528: getsockopt: connection refused

In case the master node is one of the other working nodes, I tried with all the other 2 ip addresses,but got the same result.
Should I stop the cluster before adding new nodes?
I didn't found much info on line, somebody help, please...

[OmarAyo]

Hi @Amandine-G

Thank you for using ArangoDB

In the command that you are using to add another node you are missing --starter.data-dir=path to your node

So use the following command to add more nodes

arangodb --starter.data-dir=path to your node --starter.join 65.18.27.25 --server.storage-engine=rocksdb

Note that you do not have to stop the ArangoDB Cluster to add new nodes.
Best

[Amandine-G]

Hi @OmarAyo,

Thank you for your response!
I was running the command in the data path,so I didn't use --starter.data-dir in the command at first. But I tried with your suggestion, sadly still got the same response.

I didn't find any log mentioning the new node. So what else should I check or modify?

[sleto-it]

Hi,

To reply one of your question: to add a new node it is not needed to stop the cluster

I wonder if you can share with us the commands that you have used to start the other nodes?

I assume you are starting this new node from a different machine, is that correct? Are you able to ping from this (new) machine the others?

Are you using JWT auth?

Thanks,

[Amandine-G]

Hi, I added 2 nodes to my cluster successfully! First, I excuted the command `arangodb --starter.address 65.18.27.28 --starter.data-dir=/arangodb_data` at my assuming master node.

2018/03/17 16:44:24 Starting arangodb version 0.10.4, build 553aab6 2018/03/17 16:44:24 Relaunching service with id 'bdc504a8' on 65.18.27.28:8528... 2018/03/17 16:44:24 Looking for a running instance of agent on port 8531 2018/03/17 16:44:24 Listening on 0.0.0.0:8528 (65.18.27.28:8528) 2018/03/17 16:44:24 agent seems to be running already, checking port 8531... 2018/03/17 16:44:24 agent is already running on 8531. No need to start anything. 2018/03/17 16:44:24 agent up and running (version 3.3.1). 2018/03/17 16:44:25 Just became master 2018/03/17 16:44:25 Looking for a running instance of dbserver on port 8530 2018/03/17 16:44:25 dbserver seems to be running already, checking port 8530... 2018/03/17 16:44:25 dbserver is already running on 8530. No need to start anything. 2018/03/17 16:44:25 dbserver up and running (version 3.3.1). 2018/03/17 16:44:26 Looking for a running instance of coordinator on port 8529 2018/03/17 16:44:26 coordinator seems to be running already, checking port 8529... 2018/03/17 16:44:26 coordinator is already running on 8529. No need to start anything. 2018/03/17 16:44:26 coordinator up and running (version 3.3.1). 2018/03/17 16:44:26 Your cluster can now be accessed with a browser at `http://65.18.27.28:8529` or 2018/03/17 16:44:26 using `arangosh --server.endpoint tcp://65.18.27.28:8529`.

After it writed log like the above, I retried your suggested command `arangodb --starter.data-dir=/arangodb_data --starter.join 65.18.27.28 --server.storage-engine=rocksdb `, and it printed the following log at the new node.

2018/03/17 16:44:35 Starting arangodb version 0.10.4, build 553aab6 2018/03/17 16:44:35 Contacting master http://65.18.27.28:8528... 2018/03/17 16:44:35 Waiting for 3 servers to show up... 2018/03/17 16:44:35 Serving as slave with ID '6eea5223' on :8528... 2018/03/17 16:44:35 Listening on 0.0.0.0:8528 (:8528) 2018/03/17 16:44:35 Looking for a running instance of dbserver on port 8530 2018/03/17 16:44:35 Starting dbserver on port 8530 2018/03/17 16:44:36 Looking for a running instance of coordinator on port 8529 2018/03/17 16:44:36 Starting coordinator on port 8529 2018/03/17 16:44:37 dbserver up and running (version 3.3.1). 2018/03/17 16:44:38 coordinator up and running (version 3.3.1). 2018/03/17 16:44:38 Your cluster can now be accessed with a browser at `http://65.18.27.27:8529` or 2018/03/17 16:44:38 using `arangosh --server.endpoint tcp://65.18.27.27:8529`.

and the master nodes shows

2018/03/17 16:44:35 Added new peer '6eea5223': 65.18.27.27, portOffset: 0

I don't quit understand why it has to run the first command at the master node. While excuting `arangodb` on one already running server , what it's accually doing in the background ? And for the question of @sleto-it, I am not using JWT auth. And yes, I'm starting this new node from a different machine, they ping well between each other.

[OmarAyo]

Hi @Amandine-G

I am happy to know that you have been able to add 2 more nodes.
Regarding your aforementioned misunderstandings let me do some tests and get back to you as soon as possible.

In the meanwhile, feel free to ask any other related question to this ticket.

Best,
Omar

[Amandine-G]

Hi @Omar_A, Thank you for your response. I noticed that on the 2 newly added nodes, no agents were created. So in the cluster, there are in total 3 agents, 5 Coordinators and 5 DB Servers. If I didn't take it wrong, as there is not agent created, the 2 new nodes could not become a master, is that correct? If I use `--cluster.start-agent true` to restart this 2 nodes will them function well? I will also test it.

[OmarAyo]

Hi @Amandine-G

I will divide your response into two different parts and I will give you a response for them:

• I noticed that on the 2 newly added nodes, no agents were created. So in the cluster, there are in total 3 agents, 5 Coordinators and 5 DB Servers.

Yes, with 5 nodes you have 3 Agents, 5 Coordinators and 5 DB servers. This is because by default the agency size is 3, if you want/need to increase your agency size you just have to add to your first command line
--cluster.agency-size X (for X I mean an integer, odd number) - so in your case you have to add
--cluster.agency-size 5

Please note that this command has to be run only at the first node.

You can use --cluster.start-agent true as well to have agents on the newly nodes.

• If I didn't take it wrong, as there is not agent created, the 2 new nodes could not become a master, is that correct?

Probably you are making some confusion here, please note that you can have neither master nor slave in ArangoDB Cluster deployment.
When you read the message Just became master from one of your nodes, this is referring to the Starter and not to the ArangoDB Cluster, so you do not have to worry about who is the master since you do not have a master in the ArangoDB Cluster.

In conclusion, I think that 3 Agents are enough to handle 5 nodes so no more action is needed since you have an ArangoDB Cluster with 5 nodes up and running.

For any additional question, feel free to ask.

Best.