Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add modules API for active memory defragmentation #4185

Closed
oranagra opened this issue Aug 1, 2017 · 1 comment · Fixed by #8149
Closed

Add modules API for active memory defragmentation #4185

oranagra opened this issue Aug 1, 2017 · 1 comment · Fixed by #8149

Comments

@oranagra
Copy link
Member

oranagra commented Aug 1, 2017

With current code, if anyone will enable active memory defragmentation on a server with modules who have their own data types, it can cause an overload of CPU.
The defragger will detect a fragmentation problem, and will consume a lot of CPU cycles to try and fix it, but will be unable to do so, and will continue to burn CPU cycles.

what should be done is allow modules with data types to implement callback for defragmentation work.
this should be very similar to the DEBUG DIGEST mechanism we have today.
Redis will call a module's callback for each key, and the module will iterate on it's data structure, calling redis defrag function on each pointer.
then, if the defrag function returns a non-null value, the module has to replace the pointer in it's data structures, and do any necessary adjustments on the data in the new pointer (without accessing the old pointer which was already released).

@oranagra
Copy link
Member Author

oranagra commented Nov 4, 2019

@antirez do we wanna tackle this for v6.0?
As far as i can tell, the API should include:

  • each module should announce support (yes,no,forbidden)
  • defrag callback per module type key
  • defrag callback for out of keyspace globals
  • api to defrag a pointer (which returns non-null if the pointer was moved and references should be fixed)
  • api to defrag redis primitives like dict and string
  • on each callback a module can use an API to check if time is up and exit early before the iteration on the key is complete, and resume later.
  • if the module exited early, it will be called again on the same key on the next interval.
  • modules can use static variables or private data context to be able to resume scans that exited early.

@oranagra oranagra linked a pull request Dec 8, 2020 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant