Cachelib OS memory allocation

[2matttu]

Hello, I'm part of a group of university students who are trying to modify how Cachelib allocates memory. Specifically, we are trying to make the DRAM cache be able to use memory that is mapped to devices (i.e using mmap). We were looking through the allocator/memory directory and saw that Cachelib uses contiguous memory blocks called Slabs, which are setup using SlabAllocator. However, we weren't able to pinpoint how Cachelib asks the OS for the memory region that is passed into SlabAllocator. The closest thing we found was in the AllocTestBase class in allocator/memory/tests/TestBase.h, where posix_memalign is used.

Could we be pointed to some exact locations where Cachelib interfaces with the OS to get memory? Thanks!

[leozzx]

Hi @2matttu, thanks for reaching out.

Here is where the memory is allocated and then passed into SlabAllocator: https://github.com/facebook/CacheLib/blob/main/cachelib/allocator/CacheAllocator-inl.h#L147-L158

The memory is managed by ShmManager class, here is how it asks OS for the memory internally:
https://github.com/facebook/CacheLib/blob/main/cachelib/shm/PosixShmSegment.cpp
https://github.com/facebook/CacheLib/blob/main/cachelib/shm/SysVShmSegment.cpp

[therealgymmy]

@2matttu: can you elaborate why you would like to modify DRAM cache to use memory mapped to a device. (What sort of device is this?)

If you're looking for something that works across DRAM + Flash devices, we have a HybridCache setup for that. https://cachelib.org/docs/Cache_Library_User_Guides/HybridCache

[vinser52]

@sathyaphoenix we need a way to move data between DRAM and PMEM. Imagine the case when some Item is evicted from DRAM to PMEM by Thread 1. Thread 1 should create a new Item in PMEM, copy content from the Item in DRAM to the new Item in PMEM, and update accessContainer with the new Item in PMEM. But there is possible data race if Thread 2 is calling find request for the same key. If Thread 1 hasn't updated accessContainer, Thread 2 will get Handle to the Item in DRAM and might change it while Thread 1 is doing copy operation.
To fix this issue I am thinking about the following approach. Thread 1 will create new Item in PMEM, update accessContainer and only after that start copy from the Item in DRAM to the Item in PMEM. It guarantees that if Thread 2 will call find request for the same key the Handle to the Item in PMEM will be returned. But Thread 2 should wait until Thread 1 will complete the copy operation. Therefore, I want to use wait context to make Thread 2 waiting.
Does it make sense?

[skyelves]

@sathyaphoenix we need a way to move data between DRAM and PMEM. Imagine the case when some Item is evicted from DRAM to PMEM by Thread 1. Thread 1 should create a new Item in PMEM, copy content from the Item in DRAM to the new Item in PMEM, and update accessContainer with the new Item in PMEM. But there is possible data race if Thread 2 is calling find request for the same key. If Thread 1 hasn't updated accessContainer, Thread 2 will get Handle to the Item in DRAM and might change it while Thread 1 is doing copy operation. To fix this issue I am thinking about the following approach. Thread 1 will create new Item in PMEM, update accessContainer and only after that start copy from the Item in DRAM to the Item in PMEM. It guarantees that if Thread 2 will call find request for the same key the Handle to the Item in PMEM will be returned. But Thread 2 should wait until Thread 1 will complete the copy operation. Therefore, I want to use wait context to make Thread 2 waiting. Does it make sense?

A side question: do you utilize the persistency of the PM, i.e., do you consider the item in PM persistent? If that is the case, maybe you should update accessContainer only after copying the item to PM. Otherwise, it may be fine.

[vinser52]

@skyelves Since we are using just files on PMEM-aware file system the persistence is supported out-of-the-box. I think you did mean fault tolerance. So, it is not supported because accessContainer resides in DRAM and requires serialization. We support only graceful shutdown for process restarts as it is already supported with shared memory for DRAM.

[skyelves]

Exactly! I was talking about crash consistency. I see your point. In that case, I think it's fine. Are there any advantages of using wait rather than implementing a r/w lock?
One another point, maybe it's possible to recover the accessContainer for items in PM after crashing since the items are still there. But this would incur PM allocator overhead. Anyway, it shoud be left for future.

[2matttu]

@2matttu Have you enabled usePosixForShm in the cache config?

@vinser52 Thanks on the pointer. Did you also have to set enableCachePersistency as well?

In general, we are unsure if the change from using shm_open to open has worked. We are passing shm-test-test_posix, but each of the 6 sub-tests run in 0ms, which is suspicious. How did you test/benchmark your changes?