Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add defragment support for HFE #13229

Open
wants to merge 40 commits into
base: hash-field-expiry-integ
Choose a base branch
from

Conversation

sundb
Copy link
Collaborator

@sundb sundb commented Apr 19, 2024

Background

  1. All hash objects that contain HFE are referenced by db->hexpires.
  2. All fields in a dict hash object with HFE are referenced by an ebucket.

So when we defrag the hash object or the field in a dict with HFE, we also need to update the references in them.

Interface

  1. Add a new interface ebDefragItem, which can accept a defrag callback to defrag items in ebuckets, and simultaneously update their references in the ebucket.

Mainly changes

  1. The key type of dict of hash object is no longer sds, so add new activeDefragHfieldDict() to defrag the dict instead of activeDefragSdsDict().
  2. When we defrag the dict of hash object by using dictScanDefrag(), we always set the defrag callback defragKey of dictDefragFunctions to NULL, because we can't reallocate a field with out updating it's reference in ebuckets.
    Instead, we will defrag the field of the dict and update its reference in the callback dictScanDefrag of dictScanFunction().
  3. When we defrag the hash robj with HFE, we will use ebDefragItem to defrag the robj and update the reference in db->hexpires.

Notification Support

  1. Add test to cover the notification of newly added commands.
  2. Add new notification for hpersist command.

TODO:

More test for HFE defragment.

@sundb sundb requested a review from moticless April 19, 2024 08:44
@sundb sundb marked this pull request as ready for review April 25, 2024 12:48
@sundb sundb changed the title WIP: Hash Field Expiration Defragmentation AddHash Field Expiration Defragmentation Apr 25, 2024
@sundb sundb changed the title AddHash Field Expiration Defragmentation Add defragment support for HFE Apr 25, 2024
src/ebuckets.c Outdated Show resolved Hide resolved
src/ebuckets.c Outdated Show resolved Hide resolved
src/ebuckets.c Outdated Show resolved Hide resolved
src/ebuckets.h Outdated Show resolved Hide resolved
src/server.h Outdated Show resolved Hide resolved
Copy link
Collaborator

@moticless moticless left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider avoid adding defrag logic into ebuckets.c. Instead we can expose a simplified function that merely gets pointer-to-the-pointer that points to the item. And all the logic will reside in defrag.c

Co-authored-by: Moti Cohen <moti.cohen@redis.com>
@sundb
Copy link
Collaborator Author

sundb commented Apr 27, 2024

Consider avoid adding defrag logic into ebuckets.c. Instead we can expose a simplified function that merely gets pointer-to-the-pointer that points to the item. And all the logic will reside in defrag.c

that is what i wanted to do, but then we need to expose a lot of interfaces, like ebIsList(), ebGetListPtr(), ebMarkAsList(), .etc.

@sundb
Copy link
Collaborator Author

sundb commented Apr 30, 2024

@moticless I fixed some reply schemas which failed in the schema validate CI (https://github.com/sundb/redis/actions/runs/8876700412/job/24368796489), please have a brief look.
on the other hand, i have a doubt that normally if command executes fail we will return an error message directly, rather than return an error code which is not readable, I am not sure if the current is right.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants