[SPARK-11276][CORE] SizeEstimator prevents class unloading #9244
Conversation
The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys. Which results in strong references to the Class objects. If these classes are dynamically created this prevents the corresponding ClassLoader from being GCed. Leading to PermGen exhaustion. We use a Map with WeakKeys to prevent this issue.
There was a problem hiding this comment.
Rather than involve Guava, this can just be a WeakHashMap? it has weak keys.
There was a problem hiding this comment.
A WeakHashMap would need to be synchronized and we would therefore lose concurrency.
I don't know if the SizeEstimator is used concurrently and hence felt it was safer to keep the concurrency of the original code despite the added complexity of a Guava Map.
Is it safe to drop the concurrency requirement?
EDIT: Also see this stackoverflow question
There was a problem hiding this comment.
Good call, I see why you chose this. I agree with it and LGTM.
|
Test build #1951 has finished for PR 9244 at commit
|
|
If there are no objections, I'll merge to master. This looks like a good targeted fix. |
|
merged to master |
3 participants
The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys.
Which results in strong references to the Class objects. If these classes are dynamically created
this prevents the corresponding ClassLoader from being GCed. Leading to PermGen exhaustion.
We use a Map with WeakKeys to prevent this issue.