Join GitHub today
GitHub is home to over 36 million developers working together to host and review code, manage projects, and build software together.Sign up
[SPARK-27726] [Core] Fix performance of ElementTrackingStore deletes when using InMemoryStore under high loads #24616
What changes were proposed in this pull request?
The details of the PR are explored in-depth in the sub-tasks of the umbrella jira SPARK-27726.
We see about five orders of magnitude improvement in the deletion code, which for us is the difference between a server that needs restarting daily, and one that is stable over weeks.
How was this patch tested?
Unit tests for the fire-once asynchronous code and the removeAll calls in both LevelDB and InMemoryStore are supplied. It was noted that the testing code for the LevelDB and InMemoryStore is highly repetitive, and should probably be merged, but we did not attempt that in this PR.
A version of this code was run in our production 2.3.3 and we were able to sustain higher throughput without going into GC overload (which was happening on a daily basis some weeks ago).
A version of this code was also put under a purpose-built Performance Suite of tests to verify performance under both types of Store implementations for both before and after code streams and for both total and partial delete cases (this code is not included in this PR).
I definitely need another read.
At some point I thought about allowing "unordered views", which I think would also fix the main performance problem with the deletes (which is the copy + sort operation). But I kinda like your new interface (bar some naming nits).
Yes, that's where I started too -- let me share with you why I ended up with removeAll anyway...
So, first thing I did was to remove the sort if first==last on the view, which isn't correct in general, but certainly surfaces an unordered view. The problem is that for N >> MAX, K ~= N, and the basic algorithm is still O(N^2) because we're filtering the whole list on each pass.
Having got my head on straight, the second thing I did was to move the entire delete out of the stage loop. Then I noticed that you had already done that for Tasks. But I was annoyed by the implied sort that happens anyway, so I implemented a removeIf construct (attempt unordered view part 2). This held up really well, until I performance tested with LevelDB.
LevelDB works GREAT with the original implementation. The removeIf implementation was not so wonderful when N << MAX, because then the delete is O(N) instead of O(K). I hate having to explain a performance regression on a performance PR, and that's why I moved to removeAll and separate implementations for each Store. It's actually about half the speed of the removeIf construct for InMemory, probably because of all the wrappers being built around the indexValues (stage naturalKeys). And of course, InMemoryStore still is O(N) and not O(K) on removal for N << MAX. But it's 100x faster than LevelDB, and isn't anyway a regression in behavior.
removeAll was definitely not my first go-to. :>