Garnet DB Growing Indefinitely Despite Low Data Volume After PR #1239

[wessisaac]

Describe the bug

Even after PR #1239 "Support for expired key deletion scan", we're still facing an issue where the Garnet database size grows indefinitely. Despite having less than 5MB of active data, it's consuming around 5GB of memory per day. It appears that expired or deleted records are not being physically removed from main memory, AOF, or disk (including ObjectStore and MainStore), as expected.

Steps to reproduce the bug

Git Path:
https://github.com/wesleyisaacd/GarnetRepro.git

Expected behavior

Expired and deleted data should be physically removed from main memory, AOF, and disk storage (including both the main store and object store) to prevent unnecessary memory and disk usage.

Screenshots

No response

Release version

No response

IDE

No response

OS version

No response

Additional context

No response

[badrishc]

Can you take a memory dump and share it?

[wessisaac]

Hi Badrish, please find our dump files captured at regular intervals.

Data.zip

Data2.zip

Data3.zip

Data4.zip

[hamdaankhalid]

https://github.com/microsoft/garnet/pull/1273/files

[badrishc]

Please let us know whether the linked PR resolves your issue.

[badrishc]

Please retry with the latest main as of now, thank you.

[wessisaac]

Please let us know whether the linked PR resolves your issue.

No luck, it's still growing indefinitely at around 2GB per day.

[badrishc]

Please retry with the latest main as of now, thank you.

Another PR was merged few hours back so please retry with that.

[badrishc]

If not fixed with the current latest then please attach a new dump, thank you

[wessisaac]

If not fixed with the current latest then please attach a new dump, thank you

Sure Badrish, I’ll try it out and update you.

[badrishc]

By dump, we mean a process dump of the Garnet process that will generate a .DMP file that we can analyze.