tdengine-3.0.2.4 swarm cluster (3mnode-3dnodes-3replica): 2/3 nodes restart cannot be recovered

[lazyky]

Bug Description

三个低性能 PC 的 docker swarm 集群灾备测试 ( 3mnode-3dnodes-3replica )

• 单独关闭 server-01/server-02, 插入数据，TDengine 集群运行正常, 可看到同步日志
• 同时关闭 server-01 和 server-02, 重新开机后，td-2 和 td-3 显示以下报错，TDengine 集群无法正常部署

kun_td-2.1.ktrldl0v6p8k@server-01    | 2: service ok
kun_td-2.1.ktrldl0v6p8k@server-01    | execute create dnode
kun_td-2.1.ktrldl0v6p8k@server-01    | Welcome to the TDengine Command Line Interface, Client Version:3.0.2.4
kun_td-2.1.ktrldl0v6p8k@server-01    | Copyright (c) 2022 by TDengine, all rights reserved.
kun_td-2.1.ktrldl0v6p8k@server-01    | 
kun_td-2.1.ktrldl0v6p8k@server-01    | failed to connect to server, reason: Sync not leader
kun_td-2.1.ktrldl0v6p8k@server-01    | 

To Reproduce
Steps to reproduce the behavior:

• 构建swarm集群，重启两个节点（或重启集群）

Expected Behavior
重启后，正常部署 tdengine 集群

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

• OS: ubuntu 22.04 LTS
• 8G Memory, CPU i5-1135G7, 512G SSD
• TDengine Version 3.0.2.4

Additional Context
Add any other context about the problem here.

[LZK-1994]

我这也遇到了这样的问题，目前还没解决，头疼。

[yu285]

可以升级到 3.0.3.0 问题还在的话，我可以远程看下

[yu285]

这个应该是docker自己的域名解析机制导致的。

[lazyky]

更新至3.0.3.0，重启集群仍会报错

• 重启两个节点可以正常部署集群
• 重启整个集群(三个节点)，部署时，td-2 以及 td-3 出现以下报错 failed to connect to server, reason: Sync not leader

kun_td-2.1.70jkcsew4dfy@server-01    | 2: service ok
kun_td-2.1.70jkcsew4dfy@server-01    | execute create dnode
kun_td-2.1.70jkcsew4dfy@server-01    | Welcome to the TDengine Command Line Interface, Client Version:3.0.3.0
kun_td-2.1.70jkcsew4dfy@server-01    | Copyright (c) 2022 by TDengine, all rights reserved.
kun_td-2.1.70jkcsew4dfy@server-01    | 
kun_td-2.1.70jkcsew4dfy@server-01    | failed to connect to server, reason: Sync not leader
kun_td-2.1.70jkcsew4dfy@server-01    | 

• 修改 debugFlag 为 143 获取详细报错日志，日志文件详见 td3.0.3.0_error_logs.tar.gz

# grep "failed" taosdlog.0 |head -30
03/20 16:07:05.115459 00000085 DND msg:(nil), failed to process empty msg since Mnode not found, gtid:0x:0x23bb2fe0ec000001
03/20 16:07:05.516488 00000085 DND msg:(nil), failed to process empty msg since Mnode not found, gtid:0x:0x23bb2fe0ec200002
03/20 16:07:05.916161 00000087 UTL rsetId:1 rid:3 is not there, failed to release/remove
03/20 16:07:06.317633 00000085 DND msg:(nil), failed to process empty msg since Mnode not found, gtid:0x:0x23bb2fe0ec500003
03/20 16:07:06.924429 00000088 UTL rsetId:1 rid:6 is not there, failed to release/remove
03/20 16:07:07.918663 00000085 DND msg:(nil), failed to process empty msg since Mnode not found, gtid:0x:0x23bb2fe0ecb00004
03/20 16:07:07.926204 00000087 UTL rsetId:1 rid:4 is not there, failed to release/remove
03/20 16:07:08.928624 00000088 UTL rsetId:1 rid:7 is not there, failed to release/remove
03/20 16:07:09.934541 00000087 UTL rsetId:1 rid:11 is not there, failed to release/remove
03/20 16:07:10.937082 00000088 UTL rsetId:1 rid:13 is not there, failed to release/remove
03/20 16:07:11.018418 00000089 UTL rsetId:5 rid:0 failed to remove, rid not valid
03/20 16:07:11.023809 00000087 UTL rsetId:1 rid:16 is not there, failed to release/remove
03/20 16:07:11.025009 00000114 UTL rsetId:5 rid:-1 failed to remove, rid not valid
03/20 16:07:11.026191 00000088 UTL rsetId:1 rid:20 is not there, failed to release/remove
03/20 16:07:11.119866 00000112 MND msg:0x7fb5ec005228, type:connect failed to process since Sync not leader, mnode restored:0 stopped:0, sync restored:0 role:offline, redirect numOfEps:2 inUse:0, type:connect, gtid:0x:0x23bb2fe0ed800005
03/20 16:07:11.535907 00000087 UTL rsetId:1 rid:24 is not there, failed to release/remove
03/20 16:07:11.939385 00000088 UTL rsetId:1 rid:25 is not there, failed to release/remove
03/20 16:07:11.940487 00000088 UTL rsetId:1 rid:28 is not there, failed to release/remove
03/20 16:07:11.953088 00000088 UTL rsetId:1 rid:29 is not there, failed to release/remove
03/20 16:07:11.974640 00000088 UTL rsetId:1 rid:30 is not there, failed to release/remove
03/20 16:07:12.016272 00000088 UTL rsetId:1 rid:31 is not there, failed to release/remove
03/20 16:07:12.044927 00000087 UTL rsetId:1 rid:27 is not there, failed to release/remove
03/20 16:07:12.097938 00000088 UTL rsetId:1 rid:32 is not there, failed to release/remove
03/20 16:07:12.259903 00000088 UTL rsetId:1 rid:34 is not there, failed to release/remove
03/20 16:07:12.555081 00000088 UTL rsetId:1 rid:35 is not there, failed to release/remove
03/20 16:07:12.578679 00000087 UTL rsetId:1 rid:33 is not there, failed to release/remove
03/20 16:07:12.580271 00000088 UTL rsetId:1 rid:36 is not there, failed to release/remove
03/20 16:07:12.582048 00000088 UTL rsetId:1 rid:38 is not there, failed to release/remove
03/20 16:07:12.582519 00000087 UTL rsetId:1 rid:37 is not there, failed to release/remove
03/20 16:07:12.582762 00000088 UTL rsetId:1 rid:43 is not there, failed to release/remove

• docker-compose.yml

version: "3.9"

services:
  td-1:
    build:
      dockerfile: ./docker/tdengine.Dockerfile
      args:
        TAOSD_VER: 3.0.3.0
        TZ: Asia/Shanghai
    image: localhost:5000/kun/tdengine
    networks:
      - inter
    environment: 
      TAOS_FQDN: "td-1"
      TAOS_FIRST_EP: "td-1"
      TAOS_SECOND_EP: "td-2"
    volumes:
      - taosdata-td1:/var/lib/taos/
      - taoslog-td1:/var/log/taos/
    deploy:
      placement:
        constraints:
          - node.hostname==manager
  td-2:
    image: localhost:5000/kun/tdengine
    networks:
      - inter
    environment:
      TAOS_FQDN: "td-2"
      TAOS_FIRST_EP: "td-1"
      TAOS_SECOND_EP: "td-3"
    volumes:
      - taosdata-td2:/var/lib/taos/
      - taoslog-td2:/var/log/taos/
    deploy: 
      placement:
        constraints:
          - node.hostname==server-01
  td-3:
    image: localhost:5000/kun/tdengine
    networks:
      - inter
    environment:
      TAOS_FQDN: "td-3"
      TAOS_FIRST_EP: "td-1"
      TAOS_SECOND_EP: "td-2"
      
    volumes:
      - taosdata-td3:/var/lib/taos/
      - taoslog-td3:/var/log/taos/
    deploy:
      placement:
        constraints:
          - node.hostname==server-02

  adapter:
    image: localhost:5000/kun/tdengine
    # 使用 taos -s 检测 dnodes 状态, 都为 2 时 create mnodes
    entrypoint: ["/bin/bash", "-c", "while true; \
     do \
     status=$$(taos -h td-1 -t | awk -F: '{printf $$1}' && taos -h td-2 -t | awk -F: '{printf $$1}' && taos -h td-3 -t | awk -F: '{printf $$1}'); \
     status_count=$$(awk -F2 '{print NF-1}' <<< $$status); \
     if [ $$status_count -eq 3 ]; then taos -s 'create mnode on dnode 2;create mnode on dnode 3'; break; fi; \
     done && \
     taosadapter"]
    networks:
      - inter
    environment:
      TAOS_FIRST_EP: "td-1"
      TAOS_SECOND_EP: "td-2"
    deploy:
      mode: global
      placement:
        constraints: 
          - node.role == manager

networks:
  inter:

volumes:
  taosdata-td1:
  taoslog-td1:
  taosdata-td2:
  taoslog-td2:
  taosdata-td3:
  taoslog-td3:

[lazyky]

tdengine-3.0.4.0 swarm 集群重启测试

附件为 td-1 相关日志 [taoslogs_20230417_td-3.0.4.0.tar.gz]
(https://github.com/taosdata/TDengine/files/11248944/taoslogs_20230417_td-3.0.4.0.tar.gz)

测试步骤

与之前的测试一致，td-1/2/3 为 3 个 TDengine service

• 构建 tdengine swarm 集群（3 个节点）
• 重启整个 swarm 集群（3 个节点）

service td-2/td-3 无法启动

• td-2/td-3 日志

04/17 18:10:12.873983 00000034 taos_ADAPTER info "Running in terminal." model=main
04/17 18:10:12.894274 00000034 taos_ADAPTER info "server on : 6041" model=main
2: service ok
execute create dnode
Welcome to the TDengine Command Line Interface, Client Version:3.0.4.0
Copyright (c) 2022 by TDengine, all rights reserved.

failed to connect to server, reason: Sync leader is unreachable

• 根据 /usr/bin/entrypoint.sh，节点 taos -h 检测 td-1 2: service ok，但是后续无法执行 create dnode
• TDengine集群未启动 ( td-2 和 td-3 的 taosd 未启动 ), 此时执行taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT -s "create dnode \"$FQDN:$SERVER_PORT\";" 无法连接上 TDengine
• 导致 td-2 和 td-3 报错，service td-2 和 td-3 无限重启

    while true; do
        es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check)
        echo ${es}
        if [ "${es%%:*}" -eq 2 ]; then
            echo "execute create dnode"
            taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT -s "create dnode \"$FQDN:$SERVER_PORT\";"
            break
        fi
        sleep 1s
    done

service td-1 无法启动：DND-S 相关报错

# grep RPC taosdlog.0  |grep -10 error |head -40
04/17 17:47:14.932248 00000063 RPC handle step1
04/17 17:47:14.932255 00000063 RPC conn 0x7fc371201000 start to send resp (1/2), gtid:0x:0x94638e9c8400001
04/17 17:47:14.932262 00000063 RPC rpc free cont:0x7fc371217000
04/17 17:47:14.932269 00000063 RPC DND-S conn 0x7fc371201000 start to send resp (2/2)
04/17 17:47:14.932271 00000063 RPC conn 0x7fc371201000 ref count:1
04/17 17:47:14.932275 00000063 RPC DND-S conn 0x7fc371201000 server-status-rsp is sent to 172.16.238.157:42136, local info:172.16.238.152:6030, len:146, gtid:0x:0x94638e9c8400001
04/17 17:47:14.932278 00000063 RPC conn 0x7fc371201000 ref count:2
04/17 17:47:14.932407 00000063 RPC conn 0x7fc371201000 data already was written on stream
04/17 17:47:14.932412 00000063 RPC conn 0x7fc371201000 write data out, gtid:0x:0x94638e9c8400001
04/17 17:47:14.932414 00000063 RPC conn 0x7fc371201000 ref count:1
04/17 17:47:14.932506 00000063 RPC WARN DND-S conn 0x7fc371201000 read error:EOF
04/17 17:47:14.932511 00000063 RPC conn 0x7fc371201000 to be destroyed
04/17 17:47:14.932596 00000063 RPC DND-S conn 0x7fc371201000 destroy
04/17 17:47:14.991123 00000165 RPC malloc mem:0x7fc414978720 size:83
04/17 17:47:14.991187 00000165 RPC malloc mem:0x7fc4149af0e0 size:198
04/17 17:47:14.991244 00000155 RPC rpc free cont:0x7fc414978720
04/17 17:47:14.991274 00000165 RPC malloc mem:0x7fc4149af1c0 size:200
04/17 17:47:14.991315 00000165 RPC malloc mem:0x7fc4149787e0 size:83
04/17 17:47:14.991374 00000165 RPC malloc mem:0x7fc414978840 size:83
04/17 17:47:14.991422 00000165 RPC malloc mem:0x7fc414978960 size:83
04/17 17:47:14.991756 00000155 RPC rpc free cont:0x7fc4149af0e0
--
04/17 17:47:15.537487 00000071 RPC handle step1
04/17 17:47:15.537498 00000071 RPC conn 0x7fc346001000 start to send resp (1/2), gtid:0x:0x59cf38e9c8600001
04/17 17:47:15.537515 00000071 RPC rpc free cont:0x7fc346017000
04/17 17:47:15.537619 00000062 RPC DND-S conn 0x7fc346001000 start to send resp (2/2)
04/17 17:47:15.537622 00000062 RPC conn 0x7fc346001000 ref count:1
04/17 17:47:15.537627 00000062 RPC DND-S conn 0x7fc346001000 server-status-rsp is sent to 172.16.238.162:56396, local info:172.16.238.152:6030, len:85, gtid:0x:0x59cf38e9c8600001
04/17 17:47:15.537631 00000062 RPC conn 0x7fc346001000 ref count:2
04/17 17:47:15.537727 00000062 RPC conn 0x7fc346001000 data already was written on stream
04/17 17:47:15.537733 00000062 RPC conn 0x7fc346001000 write data out, gtid:0x:0x59cf38e9c8600001
04/17 17:47:15.537736 00000062 RPC conn 0x7fc346001000 ref count:1
04/17 17:47:15.538299 00000062 RPC WARN DND-S conn 0x7fc346001000 read error:EOF
04/17 17:47:15.538304 00000062 RPC conn 0x7fc346001000 to be destroyed
04/17 17:47:15.538354 00000062 RPC DND-S conn 0x7fc346001000 destroy
04/17 17:47:15.550295 00000066 RPC new connection accepted by main server, dispatch to 1th worker-thread
04/17 17:47:15.550434 00000066 RPC success to dispatch conn to work thread
04/17 17:47:15.550449 00000063 RPC connection coming
04/17 17:47:15.550512 00000063 RPC conn 0x7fc371201000 ref count:1
04/17 17:47:15.550515 00000063 RPC DND-S handle 0x7fc371204020, conn 0x7fc371201000 created, refId:6

[lazyky]

Bug

• entrypoint.sh 的L56-L65 在第二次 TDengine 集群启动时，执行create dnode报错，docker service 重启
• 后续 taosd 没有执行，导致集群启动失败

解决方案

• 修改 TDengine 的 docker 镜像 /usr/bin/entrypoint.sh，增加检验 # check if dnode has been created
• 修改后集群可以正常启动

if [ -f "$DATA_DIR/dnode/mnodeEpSet.json" ] ||
    [ "$TAOS_FQDN" = "$FIRST_EP_HOST" ]; then
    $@
# others will first wait the first ep ready.
else
    if [ "$TAOS_FIRST_EP" = "" ]; then
        echo "run TDengine with single node."
        $@
        exit $?
    fi
    # check if dnode has been created
    if [ ! -f "$DATA_DIR/dnode/dnode.json" ]; then
        while true; do
            es=$(taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT --check)
            echo ${es}
            if [ "${es%%:*}" -eq 2 ]; then
                echo "execute create dnode"
                taos -h $FIRST_EP_HOST -P $FIRST_EP_PORT -s "create dnode \"$FQDN:$SERVER_PORT\";"
                break
            fi
            sleep 1s
        done
    fi
    $@
fi