Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Etcd registry oom exception #272

Open
ShiningSYK opened this issue Dec 28, 2023 · 7 comments
Open

Etcd registry oom exception #272

ShiningSYK opened this issue Dec 28, 2023 · 7 comments

Comments

@ShiningSYK
Copy link

dubbo版本 2.7.23

组件A(两个节点)消费组件B(两个节点),当我给组件B扩容(新增两个节点,运行服务与存量节点无异)后,组件A的一个节点输出以下报错
image
随后又持续输出以下报错
image

报错持续输出了4个小时,然后触发OOM,进程重启后恢复正常

@AlbumenJ AlbumenJ changed the title dubbo2.7.23,provider扩容后consumer连不上并且导致服务异常 Etcd registry oom exception Dec 28, 2023
@AlbumenJ AlbumenJ transferred this issue from apache/dubbo Dec 28, 2023
@AlbumenJ
Copy link
Member

This might the issue of etcd regsitry. Etcd regsitry has been moved to apache/dubbo-spi-extensions since apache dubbo 3.x.

@AlbumenJ
Copy link
Member

Can you pls reproduce it and provide the java dump?

@ShiningSYK
Copy link
Author

Can you pls reproduce it and provide the java dump?

目前无法复现

@ShiningSYK
Copy link
Author

目前情况又出现了,这次观察到的信息如下

程序运行过程中突然输出了如下信息
image
随后就一直在循环报错注册失败
image
然后我看了建立的连接,我使用了3节点的主备etcd集群,正常时它是这样的
image
随后异常,连接变成了这样
image
可以看到有对同一个etcd节点的重复链接。我不清楚是因为重复连接导致了这个问题还是这个问题导致了重复连接

@ShiningSYK
Copy link
Author

我这又有一个新思路,我这个有问题的组件,除了dubbo会跟etcd频繁交互,业务本身也会创建etcdclient直接交互,结合上面的现象,是不是两个client都对到了同一个etcd server节点上导致了这个问题,dubbo-etcd-registry这边有什么相关bug嘛

@songxiaosheng
Copy link
Member

What is the ETCD server version

@ShiningSYK
Copy link
Author

What is the ETCD server version

3.4.13

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants