taosd in K8s start failed

[liufanfandev]

tdengine 部署在 k8s 中，有一段时间网络不正常，dnode 在线的只有一个，网络恢复后有一个节点起不来，pod 一直重启。
错误日志：

容器日志

sysctl: setting key "kernel.core_pattern": Read-only file system

01/13 15:45:32.724561 00000127 TAOS_ADAPTER info "start server:00000127" model=main

01/13 15:45:32.724850 00000127 TAOS_ADAPTER info "init plugin prometheus/v1" model=plugin

01/13 15:45:32.725121 00000127 TAOS_ADAPTER info "init plugin statsd/v1" model=plugin

01/13 15:45:32.728061 00000127 TAOS_ADAPTER info "init plugin collectd/v1" model=plugin

01/13 15:45:32.729839 00000127 TAOS_ADAPTER info "init plugin influxdb/v1" model=plugin

01/13 15:45:32.730031 00000127 TAOS_ADAPTER info "init plugin node_exporter/v1" model=plugin

01/13 15:45:32.732477 00000127 TAOS_ADAPTER info "node_exporter disabled" model=NodeExporter

01/13 15:45:32.732493 00000127 TAOS_ADAPTER info "init plugin opentsdb/v1" model=plugin

01/13 15:45:32.732703 00000127 TAOS_ADAPTER info "init plugin opentsdb_telnet/v1" model=plugin

01/13 15:45:32.733803 00000127 TAOS_ADAPTER info "opentsdb_telnet disabled" model=opentsdb_telnet

01/13 15:45:32.733813 00000127 TAOS_ADAPTER info "all plugin init finish" model=plugin

01/13 15:45:32.735742 00000127 TAOS_ADAPTER info "UDP listening on "[::]:6044"" model=statsd

01/13 15:45:32.741469 00000127 TAOS_ADAPTER info "Started the statsd service on ":6044"" model=statsd

01/13 15:45:32.741514 00000127 TAOS_ADAPTER info "all plugin start finish" model=plugin

01/13 15:45:32.749078 00000127 TAOS_ADAPTER info "server on : 6041" model=main

01/13 15:45:32.957468 00000163 UTL ERROR config option:maxSQLLength, input value:10485760, out of range[65480.000000, 1048576.000000], use default value:1048576

01/13 15:45:32.957549 00000163 UTL the old unlimited para: rlim_cur=18446744073709551615, rlim_max=18446744073709551615

01/13 15:45:32.957566 00000163 UTL the new unlimited para: rlim_cur=18446744073709551615, rlim_max=18446744073709551615

01/13 15:45:32.957580 00000163 UTL _sysctl(kern_core_uses_pid) set fail: Function not implemented

01/13 15:45:32.957588 00000163 UTL The old core_uses_pid[4]: 0

01/13 15:45:32.957597 00000163 UTL _sysctl(kern_core_uses_pid) get fail: Function not implemented

01/13 15:45:32.957603 00000163 UTL The new core_uses_pid[4]: 0

01/13 15:45:32.957617 00000163 DND start to initialize TDengine

01/13 15:45:32.958582 00000163 DND step:dnode-tfile is initialized

01/13 15:45:32.958606 00000163 DND step:dnode-rpc is initialized

01/13 15:45:32.958616 00000163 UTL localEp is: datacenter-27-single-nfs-tdengine-1.datacenter-27-single-nfs-tdengine.meta42-td.svc.cluster.local:6030

01/13 15:45:32.959187 00000163 UTL WARN timezone not configured, set to system default:Asia/Shanghai (CST, +0800)

01/13 15:45:32.959300 00000163 UTL WARN locale not configured, set to system default:en_US.UTF-8

01/13 15:45:32.959335 00000163 UTL WARN charset not configured, set to system default:UTF-8

01/13 15:45:32.959390 00000163 UTL check global cfg completed

01/13 15:45:32.959400 00000163 UTL ==================================

01/13 15:45:32.959407 00000163 UTL taos config & system info:

01/13 15:45:32.959414 00000163 UTL ==================================

01/13 15:45:32.959420 00000163 UTL firstEp: datacenter-27-single-nfs-tdengine-0.datacenter-27-single-nfs-tdengine.meta42-td.svc.cluster.local:6030

01/13 15:45:32.959430 00000163 UTL secondEp: datacenter-27-single-nfs-tdengine-1.datacenter-27-single-nfs-tdengine.meta42-td.svc.cluster.local:6030

01/13 15:45:32.959436 00000163 UTL fqdn: datacenter-27-single-nfs-tdengine-1.datacenter-27-single-nfs-tdengine.meta42-td.svc.cluster.local

01/13 15:45:32.959444 00000163 UTL serverPort: 6030

01/13 15:45:32.959450 00000163 UTL configDir: /tmp/taos

01/13 15:45:32.959458 00000163 UTL logDir: /var/log/taos

01/13 15:45:32.959464 00000163 UTL scriptDir: /etc/taos

01/13 15:45:32.959472 00000163 UTL arbitrator:

01/13 15:45:32.959481 00000163 UTL numOfThreadsPerCore: 1.000000

01/13 15:45:32.959495 00000163 UTL numOfCommitThreads: 4

01/13 15:45:32.959505 00000163 UTL ratioOfQueryCores: 1.000000

01/13 15:45:32.959517 00000163 UTL maxNumOfDistinctRes: 10000000

01/13 15:45:32.959529 00000163 UTL numOfMnodes: 1

01/13 15:45:32.959541 00000163 UTL vnodeBak: 1

01/13 15:45:32.959552 00000163 UTL telemetryReporting: 1

01/13 15:45:32.959563 00000163 UTL dnodeNopLoop: 0

01/13 15:45:32.959586 00000163 UTL tcpConnTimeout: 1000(ms)

01/13 15:45:32.959598 00000163 UTL syncCheckInterval: 1500(ms)

01/13 15:45:32.959607 00000163 UTL balance: 0

01/13 15:45:32.959619 00000163 UTL balanceInterval: 300

01/13 15:45:32.959631 00000163 UTL offlineInterval: 3

01/13 15:45:32.959643 00000163 UTL keepTimeOffset: 0

01/13 15:45:32.959671 00000163 UTL role: 0

01/13 15:45:32.959684 00000163 UTL maxTmrCtrl: 512

01/13 15:45:32.959696 00000163 UTL monitorInterval: 30(s)

01/13 15:45:32.959706 00000163 UTL monitorReplica: 1

01/13 15:45:32.959717 00000163 UTL offlineThreshold: 864000(s)

01/13 15:45:32.959728 00000163 UTL rpcTimer: 300(ms)

01/13 15:45:32.959740 00000163 UTL rpcMaxTime: 600(s)

01/13 15:45:32.959752 00000163 UTL rpcForceTcp: 0

01/13 15:45:32.959764 00000163 UTL statusInterval: 1(s)

01/13 15:45:32.959776 00000163 UTL shellActivityTimer: 3(s)

01/13 15:45:32.959788 00000163 UTL minSlidingTime: 10(ms)

01/13 15:45:32.959806 00000163 UTL minIntervalTime: 1(ms)

01/13 15:45:32.959817 00000163 UTL maxStreamCompDelay: 20000(ms)

01/13 15:45:32.959826 00000163 UTL maxFirstStreamCompDelay:10000(ms)

01/13 15:45:32.959838 00000163 UTL retryStreamCompDelay: 10000(ms)

01/13 15:45:32.959849 00000163 UTL streamCompDelayRatio: 0.100000

01/13 15:45:32.959864 00000163 UTL maxVgroupsPerDb: 4

01/13 15:45:32.959876 00000163 UTL maxTablesPerVnode: 1000000

01/13 15:45:32.959888 00000163 UTL minTablesPerVnode: 1000

01/13 15:45:32.959898 00000163 UTL tableIncStepPerVnode: 1000

01/13 15:45:32.959910 00000163 UTL cache: 16(Mb)

01/13 15:45:32.959922 00000163 UTL blocks: 6

01/13 15:45:32.959934 00000163 UTL days: 10

01/13 15:45:32.959944 00000163 UTL keep: 1100

01/13 15:45:32.959955 00000163 UTL minRows: 100

01/13 15:45:32.959966 00000163 UTL maxRows: 4096

01/13 15:45:32.959977 00000163 UTL precision: 0

01/13 15:45:32.959988 00000163 UTL comp: 2

01/13 15:45:32.960000 00000163 UTL walLevel: 1

01/13 15:45:32.960012 00000163 UTL fsync: 3000

01/13 15:45:32.960024 00000163 UTL replica: 2

01/13 15:45:32.960034 00000163 UTL partitions: 4

01/13 15:45:32.960046 00000163 UTL quorum: 1

01/13 15:45:32.960058 00000163 UTL update: 0

01/13 15:45:32.960067 00000163 UTL cachelast: 0

01/13 15:45:32.960078 00000163 UTL compressMsgSize: 524288

01/13 15:45:32.960089 00000163 UTL compressColData: -1

01/13 15:45:32.960101 00000163 UTL maxSQLLength: 1048576(byte)

01/13 15:45:32.960113 00000163 UTL maxWildCardsLength: 100(byte)

01/13 15:45:32.960123 00000163 UTL maxRegexStringLen: 128(byte)

01/13 15:45:32.960135 00000163 UTL maxNumOfOrderedRes: 1000000

01/13 15:45:32.960147 00000163 UTL queryBufferSize: -1(byte)

01/13 15:45:32.960158 00000163 UTL retrieveBlockingModel: 0

01/13 15:45:32.960170 00000163 UTL keepColumnName: 1

01/13 15:45:32.960181 00000163 UTL timezone: Asia/Shanghai (CST, +0800)

01/13 15:45:32.960192 00000163 UTL locale: en_US.UTF-8

01/13 15:45:32.960201 00000163 UTL charset: UTF-8

01/13 15:45:32.960212 00000163 UTL maxShellConns: 50000

01/13 15:45:32.960221 00000163 UTL maxConnections: 50000

01/13 15:45:32.960231 00000163 UTL minimalLogDirGB: 1.000000(GB)

01/13 15:45:32.960246 00000163 UTL minimalTmpDirGB: 1.000000(GB)

01/13 15:45:32.960253 00000163 UTL minimalDataDirGB: 2.000000(GB)

01/13 15:45:32.960261 00000163 UTL mnodeEqualVnodeNum: 4

01/13 15:45:32.960272 00000163 UTL flowctrl: 1

01/13 15:45:32.960281 00000163 UTL slaveQuery: 1

01/13 15:45:32.960292 00000163 UTL adjustMaster: 1

01/13 15:45:32.960301 00000163 UTL http: 1

01/13 15:45:32.960313 00000163 UTL mqtt: 0

01/13 15:45:32.960337 00000163 UTL monitor: 1

01/13 15:45:32.960344 00000163 UTL stream: 1

01/13 15:45:32.960353 00000163 UTL topicBinaryLen: 16000

01/13 15:45:32.960362 00000163 UTL httpEnableRecordSql: 0

01/13 15:45:32.960381 00000163 UTL telegrafUseFieldNum: 0

01/13 15:45:32.960392 00000163 UTL httpMaxThreads: 100

01/13 15:45:32.960404 00000163 UTL restfulRowLimit: 10240

01/13 15:45:32.960416 00000163 UTL httpDbNameMandatory: 0

01/13 15:45:32.960428 00000163 UTL httpKeepAlive: 30000

01/13 15:45:32.960438 00000163 UTL numOfLogLines: 10000000

01/13 15:45:32.960451 00000163 UTL logKeepDays: 3

01/13 15:45:32.960462 00000163 UTL asyncLog: 1

01/13 15:45:32.960474 00000163 UTL debugFlag: 0

01/13 15:45:32.960484 00000163 UTL mDebugFlag: 131

01/13 15:45:32.960495 00000163 UTL dDebugFlag: 135

01/13 15:45:32.960512 00000163 UTL sDebugFlag: 135

01/13 15:45:32.960521 00000163 UTL wDebugFlag: 135

01/13 15:45:32.960531 00000163 UTL sdbDebugFlag: 131

01/13 15:45:32.960543 00000163 UTL rpcDebugFlag: 131

01/13 15:45:32.960555 00000163 UTL tmrDebugFlag: 131

01/13 15:45:32.960566 00000163 UTL cDebugFlag: 131

01/13 15:45:32.960578 00000163 UTL jniDebugFlag: 131

01/13 15:45:32.960587 00000163 UTL odbcDebugFlag: 131

01/13 15:45:32.960595 00000163 UTL uDebugFlag: 131

01/13 15:45:32.960601 00000163 UTL httpDebugFlag: 131

01/13 15:45:32.960608 00000163 UTL mqttDebugFlag: 131

01/13 15:45:32.960614 00000163 UTL monDebugFlag: 131

01/13 15:45:32.960621 00000163 UTL qDebugFlag: 131

01/13 15:45:32.960627 00000163 UTL vDebugFlag: 135

01/13 15:45:32.960636 00000163 UTL tsdbDebugFlag: 131

01/13 15:45:32.960646 00000163 UTL cqDebugFlag: 131

01/13 15:45:32.960658 00000163 UTL enableRecordSql: 1

01/13 15:45:32.960670 00000163 UTL enableCoreFile: 1

01/13 15:45:32.960681 00000163 UTL gitinfo: ffb4eaf

01/13 15:45:32.960693 00000163 UTL gitinfoOfInternal: NULL

01/13 15:45:32.960702 00000163 UTL buildinfo: Built at 2022-06-24 19:55

01/13 15:45:32.960713 00000163 UTL version: 2.4.0.30

01/13 15:45:32.960722 00000163 UTL maxBinaryDisplayWidth: 50

01/13 15:45:32.960733 00000163 UTL tempDir: /tmp/

01/13 15:45:32.960745 00000163 UTL tsdbMetaCompactRatio: 0

01/13 15:45:32.960758 00000163 UTL deadLockKillQuery: 1

01/13 15:45:32.960769 00000163 UTL clientMerge: 0

01/13 15:45:32.960781 00000163 UTL defaultJSONStrType: nchar

01/13 15:45:32.960792 00000163 UTL smlChildTableName:

01/13 15:45:32.960804 00000163 UTL smlTagNullName: _tag_null

01/13 15:45:32.960816 00000163 UTL walFlushSize: 1024(Mb)

01/13 15:45:32.960827 00000163 UTL lossyColumns:

01/13 15:45:32.960839 00000163 UTL fPrecision: 0.000000

01/13 15:45:32.960853 00000163 UTL dPrecision: 0.000000

01/13 15:45:32.960867 00000163 UTL maxRange: 500

01/13 15:45:32.960879 00000163 UTL range: 100

01/13 15:45:32.960892 00000163 UTL os pageSize: 4096(KB)

01/13 15:45:32.960904 00000163 UTL os openMax: 1048576

01/13 15:45:32.960915 00000163 UTL os streamMax: 16

01/13 15:45:32.960927 00000163 UTL os numOfCores: 40

01/13 15:45:32.960938 00000163 UTL os totalMemory: 128544(MB)

01/13 15:45:32.960953 00000163 UTL os sysname: Linux

01/13 15:45:32.960961 00000163 UTL os nodename: datacenter-27-single-nfs-tdengine-1

01/13 15:45:32.960968 00000163 UTL os release: 5.19.0-1.el7.elrepo.x86_64

01/13 15:45:32.960975 00000163 UTL os version: #1 SMP PREEMPT_DYNAMIC Sun Jul 31 19:04:17 EDT 2022

01/13 15:45:32.960982 00000163 UTL os machine: x86_64

01/13 15:45:32.960995 00000163 UTL dataDir: /var/lib/taos

01/13 15:45:32.961006 00000163 UTL ==================================

01/13 15:45:32.961016 00000163 DND step:dnode-globalcfg is initialized

01/13 15:45:32.961110 00000163 TFS disk /var/lib/taos is mounted to tier level 0 id 0

01/13 15:45:32.963147 00000163 UTL ==================================

01/13 15:45:32.963166 00000163 UTL os totalDisk: 3998.046875(GB)

01/13 15:45:32.963180 00000163 UTL os usedDisk: 743.954102(GB)

01/13 15:45:32.963192 00000163 UTL os availDisk: 3254.092773(GB)

01/13 15:45:32.963203 00000163 UTL ==================================

01/13 15:45:32.963216 00000163 DND dnode storage is initialized at /var/lib/taos/dnode

01/13 15:45:32.963227 00000163 DND step:dnode-storage is initialized

01/13 15:45:32.963699 00000163 DND read file /var/lib/taos/dnode/dnodeCfg.json successed

01/13 15:45:32.963887 00000163 DND dnodeId is set to 2, clusterId is set to 6d8fb722-f351-4350-94ec-e51b5b737a40

01/13 15:45:32.965231 00000163 DND successed to write /var/lib/taos/dnode/dnodeCfg.json

01/13 15:45:32.965248 00000163 DND dnode cfg is initialized

01/13 15:45:32.965254 00000163 DND dnode is dropped, exiting

01/13 15:45:32.965357 00000163 DND Failed to start TDengine, please check the log at:/var/log/taos

[yu285]

可以加微信 a15652223354 做下具体排查

[lipeidcc]

问题有解决吗？现在我遇到一样的问题 无从下手。