Skip to content

Commit

Permalink
Search: Reduce memory usage during fetch source sub phase.
Browse files Browse the repository at this point in the history
If includes or excludes are set
XContentFactory.xcontentBuilder() allocates a new
BytesStreamOutput using the default page size which is 16kb.

Can be optimized to use the length of the sourceRef because
that is the maximum possible size that the streamOutput will
use.

This redcues the amount of memory allocated for a request
that is fetching 200.000 small documents (~150 bytes each)
by about 300 MB

Close #8138
  • Loading branch information
mfussenegger authored and jpountz committed Oct 31, 2014
1 parent 7119947 commit af55479
Showing 1 changed file with 7 additions and 2 deletions.
Expand Up @@ -22,12 +22,14 @@
import com.google.common.collect.ImmutableMap;
import org.elasticsearch.ElasticsearchException;
import org.elasticsearch.common.inject.Inject;
import org.elasticsearch.common.io.stream.BytesStreamOutput;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentFactory;
import org.elasticsearch.search.SearchParseElement;
import org.elasticsearch.search.fetch.FetchSubPhase;
import org.elasticsearch.search.internal.InternalSearchHit;
import org.elasticsearch.search.internal.SearchContext;
import org.elasticsearch.search.lookup.SourceLookup;

import java.io.IOException;
import java.util.Map;
Expand Down Expand Up @@ -71,9 +73,12 @@ public void hitExecute(SearchContext context, HitContext hitContext) throws Elas
return;
}

Object value = context.lookup().source().filter(fetchSourceContext.includes(), fetchSourceContext.excludes());
SourceLookup source = context.lookup().source();
Object value = source.filter(fetchSourceContext.includes(), fetchSourceContext.excludes());
try {
XContentBuilder builder = XContentFactory.contentBuilder(context.lookup().source().sourceContentType());
final int initialCapacity = Math.min(1024, source.internalSourceRef().length());
BytesStreamOutput streamOutput = new BytesStreamOutput(initialCapacity);
XContentBuilder builder = new XContentBuilder(context.lookup().source().sourceContentType().xContent(), streamOutput);
builder.value(value);
hitContext.hit().sourceRef(builder.bytes());
} catch (IOException e) {
Expand Down

0 comments on commit af55479

Please sign in to comment.