Adding eviction policy in semantic cache class

[SHUBHAGYTA24]

What does this PR do?

The proposed changes to the semantic_cache class aim to improve its efficiency and flexibility. By introducing eviction policy FIFO, we provide users with the ability to control how the cache behaves when it reaches its maximum capacity. This is crucial for maintaining optimal cache performance and for handling situations where the available memory is constrained.

Fixes # (issue)

Who can review?

Feel free to tag members/contributors who may be interested in your PR.

@merveenoyan @stevhliu

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[stevhliu]

"I have included First In First Out (FIFO) eviction policy in the SemanticCache class to improve efficiency and flexibility. By introducing eviction policies, we provide users with the ability to control how the cache behaves when it reaches maximum capacity. This is crucial for maintaining optimal cache performance and for handling situations where memory is constrained.

Looking at the structure of the cache, the FIFO eviction policy is straightforward. Whenever a new question-answer pair is added to the cache, it's appended to the end of the lists. Thus, the oldest (first-in) items are at the beginning of the lists. When the cache reaches its maximum size and you need to evict an item, you remove (pop) the first item from each list.

However, the Least Recently Used (LRU) eviction policy is more complex because it requires knowledge of when each item in the cache was last accessed."

Reply via ReviewNB

[stevhliu]

Cool!

Is the LRU eviction policy implemented? I see that eviction_policy can be either LRU or FIFO, but in the evict function I only see FIFO implemented.

[SHUBHAGYTA24]

No, I haven't included LRU as I didn't see cache system storing order/time of access. I have mentioned that since FIFO implementation was straight forward, I implemented that only as of now.

Will implement LRU as well in future PRs.

@stevhliu

[stevhliu]

Ah ok, maybe you can mention that in the recipe to avoid confusion. Maybe something like:

"Another eviction policy is the Least Recently Used (LRU) policy, which is more complex because it requires knowledge of when each item in the cache was last accessed. However, this policy is not yet available and will be implemented later."

Could probably also remove LRU from the docstring of the evict function as well since it isn't really an option currently.

[SHUBHAGYTA24]

Sure. Done

@stevhliu

[stevhliu]

Very cool, thanks for adding to this recipe! 🤗