Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce memory usage during fetch source sub phase #8138

Closed
wants to merge 2 commits into from

Conversation

@mfussenegger
Copy link
Contributor

commented Oct 17, 2014

If includes or excludes are set
XContentFactory.xcontentBuilder() allocates a new
BytesStreamOutput using the default page size which is 16kb.

Can be optimized to use the length of the sourceRef because
that is the maximum possible size that the streamOutput will
use.

This redcues the amount of memory allocated for a request
that is fetching 200.000 small documents (~150 bytes each)
by about 300 MB

If includes or excludes are set
XContentFactory.xcontentBuilder() allocates a new
BytesStreamOutput using the default page size which is 16kb.

Can be optimized to use the length of the sourceRef because
that is the maximum possible size that the streamOutput will
use.

This redcues the amount of memory allocated for a request
that is fetching 200.000 small documents (~150 bytes each)
by about 300 MB
try {
XContentBuilder builder = XContentFactory.contentBuilder(context.lookup().source().sourceContentType());
BytesStreamOutput streamOutput = new BytesStreamOutput(source.internalSourceRef().length());

This comment has been minimized.

Copy link
@jpountz

jpountz Oct 20, 2014

Contributor

I lile it. To be even safer, maybe it would make sense to take the max with an arbitrary value: it could happen that you have very large source documents and are only interested in a small portion of them? For instance something like:

final int initialCapacity = Math.min(1024, source.internalSourceRef().length());
BytesStreamOutput streamOutput = new BytesStreamOutput(initialCapacity);

What do you think?

@jpountz

This comment has been minimized.

Copy link
Contributor

commented Oct 20, 2014

@mfussenegger This change looks good to me. I just left a minor comment, please let me know what you think.

@mfussenegger

This comment has been minimized.

Copy link
Contributor Author

commented Oct 20, 2014

Sounds good to me. I've added a fixup commit with the change. Let me know if I should squash them.

@jpountz

This comment has been minimized.

Copy link
Contributor

commented Oct 20, 2014

Unsquashed commits make the review easier, I will squash when pushing. Thanks!

@clintongormley

This comment has been minimized.

Copy link
Member

commented Oct 28, 2014

@jpountz please can you review?

@jpountz jpountz closed this in b4cad96 Oct 31, 2014
jpountz added a commit that referenced this pull request Oct 31, 2014
If includes or excludes are set
XContentFactory.xcontentBuilder() allocates a new
BytesStreamOutput using the default page size which is 16kb.

Can be optimized to use the length of the sourceRef because
that is the maximum possible size that the streamOutput will
use.

This redcues the amount of memory allocated for a request
that is fetching 200.000 small documents (~150 bytes each)
by about 300 MB

Close #8138
jpountz added a commit that referenced this pull request Oct 31, 2014
If includes or excludes are set
XContentFactory.xcontentBuilder() allocates a new
BytesStreamOutput using the default page size which is 16kb.

Can be optimized to use the length of the sourceRef because
that is the maximum possible size that the streamOutput will
use.

This redcues the amount of memory allocated for a request
that is fetching 200.000 small documents (~150 bytes each)
by about 300 MB

Close #8138
@clintongormley clintongormley changed the title reduce memory usage during fetch source sub phase Search: Reduce memory usage during fetch source sub phase Nov 3, 2014
@clintongormley clintongormley removed the review label Nov 3, 2014
@clintongormley clintongormley changed the title Search: Reduce memory usage during fetch source sub phase Reduce memory usage during fetch source sub phase Jun 7, 2015
mute pushed a commit to mute/elasticsearch that referenced this pull request Jul 29, 2015
If includes or excludes are set
XContentFactory.xcontentBuilder() allocates a new
BytesStreamOutput using the default page size which is 16kb.

Can be optimized to use the length of the sourceRef because
that is the maximum possible size that the streamOutput will
use.

This redcues the amount of memory allocated for a request
that is fetching 200.000 small documents (~150 bytes each)
by about 300 MB

Close elastic#8138
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.