Skip to content

TAOSD consums too many memory resources, failed to alter configures. #3757

@yangjnick

Description

@yangjnick

1、测试环境,三台服务器集群配置相同:
OS:CentOS Linux release 7.1.1503 (Core)
Memory: 16g
Cpu: 8
Disk: 50G
安装TDengine-server-2.0.4.0-Linux-x64.tar.gz
2、写入数据
数据库cmdata设置blocks 16,cache 64。测试程序每5分钟写入170万采集点数据
taos> show databases;
name | created_time | ntables | vgroups | replica | quorum | days | keep1,keep2,keep(D) | cache(MB) | blocks | minrows | maxrows | wallevel | fsync | comp | precision | status |

cmdata | 2020-09-25 15:20:38.150 | 1717292 | 24 | 3 | 1 | 7 | 365,365,365 | 64 | 16 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
cmanalyse | 2020-09-25 15:20:38.160 | 3 | 1 | 3 | 1 | 7 | 1095,1095,1095 | 32 | 8 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
log | 2020-09-25 11:11:57.656 | 6 | 1 | 1 | 1 | 10 | 30,30,30 | 1 | 3 | 100 | 4096 | 1 | 3000 | 2 | us | ready |
cmfre | 2020-09-25 15:20:38.155 | 0 | 0 | 3 | 1 | 7 | 30,30,30 | 32 | 8 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
Query OK, 4 row(s) in set (0.222472s)
3、问题分析:
运行三天之后taosd占用服务器内存98%,经过问题分析得出:一个节点至少需要内存166424=24576M内存,也就是24G,该问题主要是由于数据库blocks,cache设置过高,导致taosd占用内存过高。
4、尝试解决问题
尝试使用命令ALTER DATABASE db_name BLOCKS 8 修改blocks数量至一半(期望内存使用率降至12G)。
taos> show databases;
name | created_time | ntables | vgroups | replica | quorum | days | keep1,keep2,keep(D) | cache(MB) | blocks | minrows | maxrows | wallevel | fsync | comp | precision | status |

cmdata | 2020-09-25 15:20:38.150 | 1717292 | 24 | 3 | 1 | 7 | 365,365,365 | 64 | 8 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
cmanalyse | 2020-09-25 15:20:38.160 | 3 | 1 | 3 | 1 | 7 | 1095,1095,1095 | 32 | 8 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
log | 2020-09-25 11:11:57.656 | 6 | 1 | 1 | 1 | 10 | 30,30,30 | 1 | 3 | 100 | 4096 | 1 | 3000 | 2 | us | ready |
cmfre | 2020-09-25 15:20:38.155 | 0 | 0 | 3 | 1 | 7 | 30,30,30 | 32 | 8 | 100 | 4096 | 1 | 3000 | 2 | ms | ready |
Query OK, 4 row(s) in set (0.045657s)
5、观察内存使用情况、问题未解决
重启三台服务器,在不做任何查询的情况下,taos系统内存占用率仍很高,一直没有释放资源
[root@tdengine2 ~]# top
top - 16:23:34 up 3:36, 1 user, load average: 0.00, 0.26, 0.61
Tasks: 396 total, 1 running, 395 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 16255072 total, 276152 free, 15704004 used, 274916 buff/cache
KiB Swap: 8257532 total, 3206204 free, 5051328 used. 300504 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3364 root 20 0 41.623g 0.014t 320 S 1.0 94.7 11:06.35 taosd
3508 root 20 0 130300 2076 1268 R 0.3 0.0 0:00.39 top
1 root 20 0 57584 1992 112 S 0.0 0.0 0:03.37 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.09 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 0:09.42 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:0H
7 root rt 0 0 0 0 S 0.0 0.0 0:03.35 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/0
10 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcuob/1
6、尝试将blocks改为6和4仍未能释放内存,日志见附件

log.zip

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions