[Question] Does DS support dynamic expansion/reduction?

[wangmingzhiJohn]

On the index of DS website, I found this instruction called "Support dynamic expansion and reduction of master and worker".
However, in document of Expansion/Reduction, the method of achieving the goal needs to restart all services which seems like not dynamic at all.
So, my question is whether DS support dynamic way to expand or reduce nodes?

Which version of DolphinScheduler:
-[1.3.6]

[github-actions]

Hi:

• Thank you for your feedback, we have received your issue, Please wait patiently for a reply.
• In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
• If you haven't received a reply for a long time, you can subscribe to the developer's email，Mail subscription steps reference https://dolphinscheduler.apache.org/zh-cn/community/development/subscribe.html ,Then write the issue URL in the email content and send question to dev@dolphinscheduler.apache.org.

[ruanwenjun]

@wangmingzhiJohn When you add worker or master to the exist cluster, you don't need to restart the cluster.

[davidzollo]

@wangmingzhiJohn When you add worker or master to the exist cluster, you don't need to restart the cluster.

Yes, when adding, only need to start the service on the new server, and only need to stop the service of the server to be stopped

[wangmingzhiJohn]

@dailidong @ruanwenjun So, according to documents, why we need to execute stop-all.sh/start-all.sh which may influence services on other nodes

[ruanwenjun]

@wangmingzhiJohn The docs means you can use the stop-all.sh or start-all.sh to scaled-up or scaled-down the cluster.
In most case, we don't deploy all service on one machine, so you can directly use the shell. If you deploy multiple instances on one machine, you can manual start or stop the instance.

[wangmingzhiJohn]

@ruanwenjun I can't get the point. You mean these scripts can scaled-up the cluster. For example, there is 2 workers in cluster and I need to add 1. In order to let the system know that a new worker is added, I need to add a new ip to install_config.conf. And the new config file should be consistent with that on other nodes(according to doc).
As I understand it, indeed, you just need to operate on the new server to add the node into cluster and restart service. But in the start-all.sh/stop-all.sh, it will ssh to every node to restart the service whatever master or worker. I think this behavior can not be called 'dynamic scaling' because it make all services perceptive to the expansion. They all will be restart not only the service on new node

[ruanwenjun]

@wangmingzhiJohn When you add a new worker or master, the new instance will register in registry center automatic, and to exist masters will receive the added event. The script just is a normal start, stop script.

[wangmingzhiJohn]

@ruanwenjun So, actually, the exists node's service don't need to update their install_config.conf as they can know the new node by zookeeper automaticly, is that correct?

[ruanwenjun]

@wangmingzhiJohn Yes, and other registry center will be supported in the future, currently only zookeeper.

[wangmingzhiJohn]

@ruanwenjun In another way, these specific ips in install_config.conf is not necessary if I setup every service on every node manually by dolphinscheduler-daemon.sh?

[wangmingzhiJohn]

Or except zookeeper's ip:port which is necessary.