[Feature] Make StarRocks support FQDN #5127

waittttting · 2022-04-15T11:53:18Z

What type of PR is this：

bug
feature
enhancement
others

Which issues of this PR fixes ：

Problem Summary(Required):

StarRocks only supported IP before. Some users' network environments need to support FQDN.
You can refer to the following documentation to use FQDN in your StarRocks cluster
First, you need to configure all the FQDN information of the cluster in each machine's /etc/hosts file.
If you want to use FQDN to create a new cluster, you should do nothing, this feature will make cluster use fqdn automatically
And if you want to upgrade an old cluster, please follow the next steps.

Be change ip to fqdn

BE Reboot directly after upgrading to the version that contains the feature
FE Execute SQL
alter system modify backend host "192.1.1.0" to "testFqdn"

Fe upgrade ip to fqdn

FE cluster upgrade with less than three followers is not supported
Change follower node first, then master node

How to follower's ip to fqdn?

stop the node you want to change the IP to FQDN, wait for the alive state to be false
Execute the following SQL
alter system modify frontend host "127.1.1.0" to "sandbox16"
After the FE is upgraded to the version that includes the feature
restart the FE with sh bin/start_fe.sh --host_type=fqdn --daemon and wait for the FE alive state to be true

How to master's ip to fqdn?

When all the follower besides of the master in the cluster are upgraded, shut down the master and wait for the new master to be elected, and then restart the original master, wait it to be a follower and alive.
Then the new follower could be processed according to the above to upgrade ip to fqdn

Other:

this feature can help you start fe by using specified host type，just like sh bin/start_fe.sh --host_type=fqdn sh bin/start_fe.sh --host_type=ip

padmejin · 2022-04-16T05:48:02Z

Please add a more detailed description. This is an obvious change for the users, that is to say, all our users will notice that after this version released, their FQDN can be directly shown in the "show backends" result or on our dashboard, and that casually changing their FQDN will probably ends in a chaos for the whole cluster. Therefore we should explain our designs and trade-offs as clear as possible.

be/src/service/backend_options.cpp

be/src/util/brpc_stub_cache.h

fe/fe-core/src/main/java/com/starrocks/system/SystemInfoService.java

be/src/util/brpc_stub_cache.h

fe/fe-core/src/main/jflex/sql_scanner.flex

be/src/common/config.h

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java

chaoyli · 2022-04-17T13:55:40Z

Create a issue related to this pull request.

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java

fe/fe-core/src/main/java/com/starrocks/system/SystemInfoService.java

gengjun-git · 2022-04-22T08:20:47Z

Remember to add user guide to https://github.com/StarRocks/docs.zh-cn

be/src/util/brpc_stub_cache.h

fe/fe-core/src/main/java/com/starrocks/service/FrontendOptions.java

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java

chaoyli · 2022-04-24T05:34:41Z

Why we must restart FE to change the IP:port to 'FQDN:port'?
It's not very friendly to users. The cluster online should avoid the restart as far as possible.

fe/fe-core/src/main/cup/sql_parser.cup

waittttting · 2022-04-24T15:25:10Z

Why we must restart FE to change the IP:port to 'FQDN:port'? It's not very friendly to users. The cluster online should avoid the restart as far as possible.

As stated in the PR description, if the host is changed without restarting the cluster, it is a process with very many intermediate states, and the failure of any one step does not guarantee the consistency of the change, and the user may end up having to restart the cluster anyway

fe/fe-core/src/main/java/com/starrocks/system/Frontend.java

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java

fe/fe-core/src/main/java/com/starrocks/common/util/NetUtils.java

fe/fe-core/src/main/java/com/starrocks/server/GlobalStateMgr.java

waittttting · 2022-06-01T11:04:34Z

run starrocks_fe_unittest

wanpengfei-git · 2022-06-01T11:11:21Z

[FE PR Coverage check]

😍 pass : 221 / 275 (80.36%)

file detail

	path	covered line	new line	coverage
🔵	com/starrocks/alter/Alter.java	0	1	00.00%
🔵	com/starrocks/StarRocksFE.java	0	3	00.00%
🔵	com/starrocks/qe/DdlExecutor.java	0	1	00.00%
🔵	com/starrocks/alter/SystemHandler.java	4	7	57.14%
🔵	com/starrocks/alter/SchemaChangeHandler.java	4	7	57.14%
🔵	com/starrocks/persist/EditLog.java	3	5	60.00%
🔵	com/starrocks/server/NodeMgr.java	29	42	69.05%
🔵	com/starrocks/service/FrontendOptions.java	60	76	78.95%
🔵	com/starrocks/server/GlobalStateMgr.java	4	5	80.00%
🔵	com/starrocks/sql/analyzer/AlterSystemStmtAnalyzer.java	11	13	84.62%
🔵	com/starrocks/common/proc/FrontendsProcNode.java	7	8	87.50%
🔵	com/starrocks/system/SystemInfoService.java	42	48	87.50%
🔵	com/starrocks/analysis/ModifyFrontendAddressClause.java	8	9	88.89%
🔵	com/starrocks/analysis/ModifyBackendAddressClause.java	8	9	88.89%
🔵	com/starrocks/sql/parser/AstBuilder.java	6	6	100.00%
🔵	com/starrocks/common/proc/BackendsProcDir.java	1	1	100.00%
🔵	com/starrocks/sql/ast/AstVisitor.java	2	2	100.00%
🔵	com/starrocks/persist/Storage.java	11	11	100.00%
🔵	com/starrocks/system/Backend.java	2	2	100.00%
🔵	com/starrocks/catalog/BrokerMgr.java	1	1	100.00%
🔵	com/starrocks/common/util/NetUtils.java	11	11	100.00%
🔵	com/starrocks/system/Frontend.java	3	3	100.00%
🔵	com/starrocks/ha/BDBHA.java	3	3	100.00%
🔵	com/starrocks/alter/MaterializedViewHandler.java	1	1	100.00%

This reverts commit aed944f.

StarRocks only supported IP before. Some users' network environments need to support FQDN. You can refer to the following documentation to use FQDN in your StarRocks cluster First, you need to configure all the FQDN information of the cluster in each machine's /etc/hosts file. If you want to use FQDN to create a new cluster, you should do nothing, this feature will make cluster use fqdn automatically And if you want to upgrade an old cluster, please follow the next steps. ### **Be change ip to fqdn** 1. BE Reboot directly after upgrading to the version that contains the feature 2. FE Execute SQL alter system modify backend host "192.1.1.0" to "testFqdn" ### **Fe upgrade ip to fqdn** - FE cluster upgrade with less than three followers is not supported - Change follower node first, then master node **How to follower's ip to fqdn?** 1. stop the node you want to change the IP to FQDN, wait for the alive state to be false 2. Execute the following SQL alter system modify frontend host "127.1.1.0" to "sandbox16" 3. After the FE is upgraded to the version that includes the feature 4. restart the FE with `sh bin/start_fe.sh --host_type=fqdn` --daemon and wait for the FE alive state to be true **How to master's ip to fqdn?** When all the follower besides of the master in the cluster are upgraded, shut down the master and wait for the new master to be elected, and then restart the original master, wait it to be a follower and alive. Then the new follower could be processed according to the above to upgrade ip to fqdn ## Other: this feature can help you start fe by using specified host type，just like `sh bin/start_fe.sh --host_type=fqdn` `sh bin/start_fe.sh --host_type=ip`

…Rocks#6945) This reverts commit aed944f.

Signed-off-by: EsoragotoSpirit <wanglichen@starrocks.com> (cherry picked from commit c9f5695) Co-authored-by: 絵空事スピリット <wanglichen@starrocks.com>

waittttting changed the title ~~Main fqdn~~ Make StarRocks support FQDN Apr 16, 2022

padmejin reviewed Apr 16, 2022

View reviewed changes

murphyatwork reviewed Apr 17, 2022

View reviewed changes

fe/fe-core/src/main/jflex/sql_scanner.flex Outdated Show resolved Hide resolved

chaoyli requested changes Apr 17, 2022

View reviewed changes

be/src/common/config.h Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java Outdated Show resolved Hide resolved

chaoyli requested changes Apr 17, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java Outdated Show resolved Hide resolved

waittttting force-pushed the main-fqdn branch 2 times, most recently from 1c3a115 to 3cccdb1 Compare April 20, 2022 12:46

padmejin reviewed Apr 22, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/system/SystemInfoService.java Outdated Show resolved Hide resolved

gengjun-git reviewed Apr 22, 2022

View reviewed changes

be/src/util/brpc_stub_cache.h Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/service/FrontendOptions.java Outdated Show resolved Hide resolved

waittttting force-pushed the main-fqdn branch from f9368bf to e9e1505 Compare April 23, 2022 06:30

chaoyli requested changes Apr 24, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java Outdated Show resolved Hide resolved

meegoo reviewed Apr 24, 2022

View reviewed changes

fe/fe-core/src/main/cup/sql_parser.cup Outdated Show resolved Hide resolved

waittttting force-pushed the main-fqdn branch 2 times, most recently from 8fbf157 to b22e52d Compare April 26, 2022 12:17

chaoyli changed the title ~~Make StarRocks support FQDN~~ [Feature] Make StarRocks support FQDN Apr 27, 2022

waittttting force-pushed the main-fqdn branch 4 times, most recently from c9e474c to 855062a Compare April 29, 2022 03:15

gengjun-git reviewed Apr 29, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/system/Frontend.java Outdated Show resolved Hide resolved

padmejin reviewed Apr 29, 2022

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/analysis/UpdateBackendAddressClause.java Outdated Show resolved Hide resolved

fe/fe-core/src/main/java/com/starrocks/common/util/NetUtils.java Outdated Show resolved Hide resolved

gengjun-git reviewed Apr 29, 2022

View reviewed changes

waittttting force-pushed the main-fqdn branch 3 times, most recently from 8aeda1d to 5b5068f Compare May 5, 2022 02:22

waittttting added 12 commits June 1, 2022 17:32

Fe support use SQL modify hostAddress without delete meta data

99e97b7

Add test case & Modify issues in comments

00219bf

Adapting to the new parser & Add test case

ee8a2d7

Update modify backend host SQL & Backend support fqdn without use config

e8d28ad

Modify issues in comments

227b9b6

Resolve conflicts with existing code

fd02aa9

Removed enable_fqdn from fe configuration file & fix bug

2208291

Resolve conflicts

8886841

Support for choosing whether to use ip or fqdn to start fe

f51ede8

Add case & Modify issues in comments

8848de4

Resolve conflicts

529c803

Modify issues in comments

6e23e6c

waittttting dismissed stale reviews from chaoyli and gengjun-git via 6e23e6c June 1, 2022 09:36

waittttting force-pushed the main-fqdn branch from afe4be8 to 6e23e6c Compare June 1, 2022 09:36

chaoyli approved these changes Jun 6, 2022

View reviewed changes

gengjun-git approved these changes Jun 6, 2022

View reviewed changes

gengjun-git merged commit aed944f into StarRocks:main Jun 6, 2022

waittttting deleted the main-fqdn branch June 6, 2022 08:26

waittttting restored the main-fqdn branch June 7, 2022 04:43

waittttting added a commit that referenced this pull request Jun 7, 2022

Revert "[Feature] Make StarRocks support FQDN (#5127)"

68ee4c3

This reverts commit aed944f.

waittttting mentioned this pull request Jun 7, 2022

Revert "[Feature] Make StarRocks support FQDN" #6945

Merged

chaoyli pushed a commit that referenced this pull request Jun 7, 2022

Revert "[Feature] Make StarRocks support FQDN (#5127)" (#6945)

93efb9a

This reverts commit aed944f.

waittttting mentioned this pull request Jun 7, 2022

Revert "[Feature] Make StarRocks support FQDN (#5127)" (#6945) #6946

Merged

imay pushed a commit that referenced this pull request Jun 7, 2022

Revert "[Feature] Make StarRocks support FQDN (#5127)" (#6945) (#6946)

6d7e275

This reverts commit aed944f.

abc982627271 pushed a commit to abc982627271/starrocks that referenced this pull request Jun 22, 2022

Revert "[Feature] Make StarRocks support FQDN (StarRocks#5127)" (Star…

8cb32a4

…Rocks#6945) This reverts commit aed944f.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Make StarRocks support FQDN #5127

[Feature] Make StarRocks support FQDN #5127

waittttting commented Apr 15, 2022 •

edited

Loading

padmejin commented Apr 16, 2022

chaoyli commented Apr 17, 2022

gengjun-git commented Apr 22, 2022

chaoyli commented Apr 24, 2022

waittttting commented Apr 24, 2022

waittttting commented Jun 1, 2022

wanpengfei-git commented Jun 1, 2022

[Feature] Make StarRocks support FQDN #5127

[Feature] Make StarRocks support FQDN #5127

Conversation

waittttting commented Apr 15, 2022 • edited Loading

What type of PR is this：

Which issues of this PR fixes ：

Problem Summary(Required):

Be change ip to fqdn

Fe upgrade ip to fqdn

Other:

padmejin commented Apr 16, 2022

chaoyli commented Apr 17, 2022

gengjun-git commented Apr 22, 2022

chaoyli commented Apr 24, 2022

waittttting commented Apr 24, 2022

waittttting commented Jun 1, 2022

wanpengfei-git commented Jun 1, 2022

[FE PR Coverage check]

file detail

waittttting commented Apr 15, 2022 •

edited

Loading