-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-14289][WIP] Support multiple eviction strategies for cached RDD partitions #12162
Conversation
Can one of the admins verify this patch? |
Thanks for the pull request. Is this actually motivated by a real use case, or just doing it because it might be good to support more than one policy? |
@rxin The use case that motivate this is about below.
This PR can be separated to below sub-task.
|
@Earne is something relevant merged in spark 2.0.1, do we have FIFO eviction policy?. |
This branch appears to be incomplete. The configuration parameter |
@Earne, is it still active and any opinion on the comments above? Otherwise, I will propose to close this. |
I'm working on my thesis to improve cache management systems. But i don't know anything about Spark! I found this program and I don't know how to even run it. If possible, can someone refer to me a video or steps in order to run this file? If i can see a sample implementation of LRU and know how it's made step by step then I'll be able to implement my own algorithm. I would be very grateful if someone can offer their help! |
As best I can tell, the code that was pushed here is incomplete. However, Spark's default cache eviction policy is LRU. You can find the code which performs eviction here. It basically just works by storing all the data in a |
What changes were proposed in this pull request?
Currently, LRU is the only eviction strategy for cached RDD partitions in Spark.
This pull request will refactor and add support to multiple eviction strategies, such as FIFO, LFU(WIP), LCS(WIP).
How was this patch tested?
Manual test by set "spark.memory.entryEvictionPolicy" to LRU(default), FIFO or LCS.