[Bug] FE 频繁打满64G内存导致宕机，集群上只有Broker Load在定时执行，过一段时间内存就满了

[DA1OOO]

Search before asking

• I had searched in the issues and found no similar issues.

Version

版本 2.0.1.1 release

What's Wrong?

下图是内存使用情况，内存无法回收，每次需要重启，过一段时间又满了：

版本 2.0.1.1 release
JVM -xmx64g
宕机前gc日志：

宕机前fe.log:

来自该机器的sql只有broker load，show load，show partitions，drop partition，add partition这几类。

What You Expected?

什么原因导致FE 宕机，应该不是64g内存也不够吧

How to Reproduce?

No response

Anything Else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zengxiangqi1031]

遇到同样的问题，doris 2.0.2 release

[DA1OOO]

JAVA_OPTS="-Djavax.security.auth.useSubjectCredsOnly=false -Xss4m -Xmx64g -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$CUR_DATE"

[DA1OOO]

只改了-xmx大小 其他都是默认JVM配置

[liugddx]

Using G1GC

[DA1OOO]

Using G1GC

thanks, i will try

[DA1OOO]

btw, i used broker 2.0.2, not 2.0.1.1.

[DA1OOO]

[liugddx]

https://doris.apache.org/zh-CN/docs/1.2/admin-manual/query-profile?_highlight=profile#%E5%90%8D%E8%AF%8D%E8%A7%A3%E9%87%8A

Maybe you need to turn off the global profile SET [GLOBAL] enable_profile=false;

[DA1OOO]

After reviewing the source code, the default max_query_profile_num seems to be 100, so it would't keep pushing profile into memory?

[liugddx]

After reviewing the source code, the default max_query_profile_num seems to be 100, so it would't keep pushing profile into memory?

I don’t have a detailed understanding yet. You can continue to follow or provide more detailed log information.

[DA1OOO]

After restart fe and SET [GLOBAL] enable_profile=false:


I have a broker load task running from 11:34 to 11:36, which is when the memory is rapidly increasing.

[liugddx]

Has this memory problem affected usage? In addition, will the memory be lost by gc?

[DA1OOO]

I need to observer the change of memory after closeing the enable_profile. But before closing it, memory just lost a little by gc, after the memory reaches the maximum value set by -xmx, FE will stop serving.

[wj215318]

I need to observer the change of memory after closeing the enable_profile. But before closing it, memory just lost a little by gc, after the memory reaches the maximum value set by -xmx, FE will stop serving.

how about fe memory after closeing the enable_profile,thanks

[DA1OOO]

It seems become normal now. Maybe remove profile have some bug. @wj215318

[wj215318]

It seems become normal now. Maybe remove profile have some bug. @wj215318

We have encountered the same problem.and now we also closed the profile.yestoday wo dump the jvm data,DBA is analyzing

[DA1OOO]

Due to the impact of dumping on the normal use of the cluster, we did not dump the JVM data. If you discover anything after dumping, please share the specific situation here. @wj215318 Thanks!

[DA1OOO]

@wj215318 btw, 2.0.2 release don't have this problem.

[ziyanTOP]

一样的问题，minor gc的频率跟不上老年代增长的速度，最后三个fe节点全部查询排队超时卡死宕机，建议用prometheus+grafana监控fe的JVM，看看到底问题出在哪，顺便改下你的参数，年轻代等于老年代的1/3，并且不要用-XX:NewRatio=3这种，而是固定设置成-Xmn16G，打开CMS的并行重标记，不然minor gc那点时间这么多内存根本标记不完，然后调低CMS初始化时的内存占比，80%太靠后了，可能gc没完成服务就down了，可以改成60或者65，实测有效，我的集群调整完至今没有fe宕机

[zhbdesign]

一样的问题，minor gc的频率跟不上老年代增长的速度，最后三个fe节点全部查询排队超时卡死宕机，建议用prometheus+grafana监控fe的JVM，看看到底问题出在哪，顺便改下你的参数，年轻代等于老年代的1/3，并且不要用-XX:NewRatio=3这种，而是固定设置成-Xmn16G，打开CMS的并行重标记，不然minor gc那点时间这么多内存根本标记不完，然后调低CMS初始化时的内存占比，80%太靠后了，可能gc没完成服务就down了，可以改成60或者65，实测有效，我的集群调整完至今没有fe宕机

修改后的启动参数可以分享下

[ziyanTOP]

JAVA_OPTS="-server -Xmx64g -Xmn16g -Xms32g -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=15 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:+CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=65 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE" @zhbdesign 具体内存大小根据机器的实际值来设置

[DA1OOO]

用了G1回收器 调大JVM内存后。目前正常。

还是不理解为什么内存增速这么快。

[ihadoop]

dump下来的文件可以上传上来