Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Storaged memory continues to grow until the leader change or OOM #3779

Closed
carrey-feng opened this issue Jan 21, 2022 · 4 comments
Closed

Storaged memory continues to grow until the leader change or OOM #3779

carrey-feng opened this issue Jan 21, 2022 · 4 comments
Labels
type/bug Type: something is unexpected

Comments

@carrey-feng
Copy link
Contributor

carrey-feng commented Jan 21, 2022

Describe the bug (required)
All storageds memory continues to grow until someone OOM or leader changes by running balance leader command when leader is unblanced in cluster.
I used jemalloc to dump malloc, I found many "nebula::ConstantExpression::clone" before OOM. and it disappeared or became less after OOM or Leader changed.

I found that "nebula::ConstantExpression::clone" was called by fetching default value of edge property.
I do not know if it leaked and why it became less after Leader changed.
Can you give me some ideas on how to check. thanks

Your Environments (required)

How To Reproduce(required)

Steps to reproduce the behavior:

  1. Step 1
  2. Step 2
  3. Step 3

Expected behavior

Additional context

image

08:35 jeprof top
image

08:36 jeprof top
image

08:37 jeprof top
image

08:35 call graph
210835.pdf

image

@carrey-feng carrey-feng added the type/bug Type: something is unexpected label Jan 21, 2022
@carrey-feng carrey-feng changed the title Storaged memory continues to grow until the leader changes or OOM Storaged memory continues to grow until the leader change or OOM Jan 21, 2022
@cangfengzhs
Copy link
Contributor

Yes, this is a serious bug.And you create a tag/edge with a default value, right? We have found and fixed it. Look at #3601.

@carrey-feng
Copy link
Contributor Author

Yes, this is a serious bug.And you create a tag/edge with a default value, right? We have found and fixed it. Look at #3601.

It is great ! I have deleted edge default value, memory usage is very flat. Thanks!

Another problem that #3729 was create at last week, there is anyone to review it so far. I think that it is import to multi-tenant cluster. It can prevent one user's mistake from affecting all other users. Could you review it ? Thanks !

@Sophie-Xie
Copy link
Contributor

Yes, this is a serious bug.And you create a tag/edge with a default value, right? We have found and fixed it. Look at #3601.

It is great ! I have deleted edge default value, memory usage is very flat. Thanks!

Another problem that #3729 was create at last week, there is anyone to review it so far. I think that it is import to multi-tenant cluster. It can prevent one user's mistake from affecting all other users. Could you review it ? Thanks !

Thanks. I just assigned two reviewers.

@wey-gu
Copy link
Contributor

wey-gu commented Jan 24, 2022

Thanks @a516072575 , great option introduced.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type/bug Type: something is unexpected
Projects
None yet
Development

No branches or pull requests

4 participants