Internal: Filter cache size limit not honored for 32GB or over

[danp60]

Hi,

We are running an elasticsearch 1.1.1 6 node cluster with 256GB of ram, and using 96GB JVM heap sizes. I've noticed that when I set the filter cache size to 32GB or over with this command:

curl -XPUT "http://localhost:9200/_cluster/settings" -d'
{
    "transient" : {
       "indices.cache.filter.size" : "50%"
    }
}'

The field cache size keeps growing above and beyond the indicated limit. The relevant node stats show that the filter cache size is about 69GB in size, which is over the configured limit of 48GB

"filter_cache" : {
    "memory_size_in_bytes" : 74550217274,
    "evictions" : 8665179
},

I've enable debug logging on the node itself and it looks like the cache itself is getting created with the correct values:

[2014-05-21 00:31:57,215][DEBUG][indices.cache.filter     ] [ess02-006] using [node] weighted filter cache with size [50%], actual_size [47.9gb], expire [null], clean_interval [1m]

Whats strange is that when I set the limit to 31.9GB, the limit is enforced, which leads me to believe there is some sort of overflow going on.

Thanks,
Daniel

[danp60]

Hi,
I dug a little deeper into the caching logic, and I think I have found the root cause. The class IndicesFilterCache sets concurrencyLevel to a hardcoded 16:

private void buildCache() {
    CacheBuilder<WeightedFilterCache.FilterCacheKey, DocIdSet> cacheBuilder = CacheBuilder.newBuilder()
            .removalListener(this)
            .maximumWeight(sizeInBytes).weigher(new WeightedFilterCache.FilterCacheValueWeigher());

    // defaults to 4, but this is a busy map for all indices, increase it a bit
    cacheBuilder.concurrencyLevel(16);

    if (expire != null) {
        cacheBuilder.expireAfterAccess(expire.millis(), TimeUnit.MILLISECONDS);
    }

    cache = cacheBuilder.build();
}

https://github.com/elasticsearch/elasticsearch/blob/9ed34b5a9e9769b1264bf04d9b9a674794515bc6/src/main/java/org/elasticsearch/indices/cache/filter/IndicesFilterCache.java#L116

In the Guava libraries, the eviction code is as follows:

void evictEntries() {
    if (!map.evictsBySize()) {
        return;
    }

    drainRecencyQueue();
    while (totalWeight > maxSegmentWeight) {
        ReferenceEntry<K, V> e = getNextEvictable();
        if (!removeEntry(e, e.getHash(), RemovalCause.SIZE)) {
            throw new AssertionError();
        }
    }
}

https://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/cache/LocalCache.java#2659

Since totalWeight is an int and maxSegmentWeight is a long set to maxWeight / concurrencyLevel, when maxWeight is 32GB or above, then the value of maxSegmentWeight will be set to above the maximum value of int and the check

while (totalWeight > maxSegmentWeight) {

will always fail.

[jpountz]

Wow, good catch! I think it would make sense to file a bug to Guava?

[nik9000]

Wow, good catch! I think it would make sense to file a bug to Guava?

Indeed!

I'd file that with Guava but also clamp the size of the cache in Elasticsearch to 32GB - 1 for the time being.

As an aside I imagine 96GB heaps cause super long pause times on hot spot.

[jpountz]

+1

[nik9000]

I've got the code open and have a few free moments so I can work on it if no one else wants it.

[jpountz]

That works for me, feel free to ping me when it's ready and you want a review.

[kimchy]

Wow, good catch! I think it would make sense to file a bug to Guava?

Huge ++!. @danp60 when you file the bug in guava, can you link back to it here?

[nik9000]

I imagine you've already realized it but the work around is to force the cache size under 32GB.

[jpountz]

Indeed. I think that's not too bad a workaround though since I would expect such a large filter cache to be quite wasteful compared to leaving the memory to the operating system so that it can do a better job with the filesystem cache.

[danp60]

@kimchy I've filed the guava bug here: https://code.google.com/p/guava-libraries/issues/detail?id=1761&colspec=ID%20Type%20Status%20Package%20Summary

[jpountz]

@danp60 Thanks!

[jpountz]

The bug has been fixed upstream.