Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase default heap size to 2g #18311

Merged
merged 2 commits into from May 14, 2016

Conversation

Projects
None yet
9 participants
@jasontedor
Copy link
Member

commented May 13, 2016

Benchmarking with the geonames data set shows that Elasticsearch truly
needs 2g of heap or the heap is a bottleneck for indexing rates. If our
goal is to satisfy the out-of-the-box-experience, we should prioritize
the out-of-the-box performance experience. Thus, the default heap should
be 2g until smaller heaps are not the bottleneck for indexing small data
sets.

Relates #16334, relates #17686, relates #18309

Increase default heap size to 2g
Benchmarking with the geonames data set shows that Elasticsearch truly
needs 2g of heap or the heap is a bottleneck for indexing rates. If our
goal is to satisfy the out-of-the-box-experience, we should prioritize
the out-of-the-box performance experience. Thus, the default heap should
be 2g until smaller heaps are not the bottleneck for indexing small data
sets.
@rmuir

This comment has been minimized.

Copy link
Contributor

commented May 13, 2016

+1, it makes the defaults honest. 512MB is just not enough.

@dadoonet

This comment has been minimized.

Copy link
Member

commented May 13, 2016

+1

@s1monw

This comment has been minimized.

Copy link
Contributor

commented May 13, 2016

lets do it! +1

@s1monw

This comment has been minimized.

Copy link
Contributor

commented May 13, 2016

since we now increase the heap size should we add benchmarks with lower heap size to track if stuff gets better / worse?

@danielmitterdorfer

This comment has been minimized.

Copy link
Member

commented May 13, 2016

@s1monw We can do this. I can pin one setup to 1gb but I'd say we don't include the (moving) default heap size in the benchmarks (i.e. now the 2gb setting) from now on. Wdyt?

@s1monw

This comment has been minimized.

Copy link
Contributor

commented May 13, 2016

yeah why not

@danielmitterdorfer

This comment has been minimized.

Copy link
Member

commented May 13, 2016

Provided nobody else objects, the default setup will run the benchmark candidate with 1GB heap as of the next nightly run (i.e. tomorrow).

danielmitterdorfer added a commit to elastic/rally that referenced this pull request May 13, 2016

@jasontedor

This comment has been minimized.

Copy link
Member Author

commented May 13, 2016

I think that we should be benchmarking the 2g configuration because below that we are just hamstringing the benchmarks by the heap. Also, I've come to the view that we should be benchmarking the defaults so we can watch for regressions there. If this is the default, that's what users will be running if they do not change any settings. We see lots of benchmarks in the wild that do not modify the out-of-the-box settings, for better or for worse.

@mikemccand

This comment has been minimized.

Copy link
Contributor

commented May 13, 2016

I think it's vital that we DO benchmark at the default settings: many users will do just that and draw conclusions. In fact, I think it's the most important curve in the nightly benchmarks ;)

We can also benchmark at other fixed heap sizes so we know the impact, of either too little or too big heap sizes.

@danielmitterdorfer

This comment has been minimized.

Copy link
Member

commented May 13, 2016

Ok, then I'll revert.

@s1monw

This comment has been minimized.

Copy link
Contributor

commented May 13, 2016

I misunderstood - of course we have to benchmark defaults. I though you are saying you are not adding 2GB explicitly

Restore unequal min and max default heap
After considerable discussion, we have elected to set the default min
heap to 256m and the default max heap to 2g. This is to balance the
desire for a good out-of-the-box performance experience (default max
heap of 2g) with a good out-of-the-box experience running on machines
with limited resources or running multiple instances on a single modern
developer laptop (default min heap of 1g).
@jasontedor

This comment has been minimized.

Copy link
Member Author

commented May 13, 2016

@clintongormley I pushed bc46517 to set unequal defaults of 256m and 2g.

@jasontedor jasontedor removed the discuss label May 13, 2016

@jasontedor jasontedor merged commit 10a881d into elastic:master May 14, 2016

1 check passed

CLA Commit author is a member of Elasticsearch
Details

@jasontedor jasontedor deleted the jasontedor:heap-2g branch May 14, 2016

@djschny

This comment has been minimized.

Copy link
Contributor

commented May 17, 2016

While the change itself is obviously trivial, this change is a big deal IMO. There are tons of users who use elasticsearch in limited resource environments. Before they could use the defaults and not worry about consuming too many resources.

For these reasons I feel this should be labelled as a breaking change.

@clintongormley

This comment has been minimized.

Copy link
Member

commented May 18, 2016

@djschny nobody should be using the default heap size in production. In fact, we fail if min heap != max heap in production mode

@djschny

This comment has been minimized.

Copy link
Contributor

commented May 18, 2016

@clintongormley what is this "production mode" spoken of? If it is the startup checks that happen when not bound to localhost, then those did not exist in 2.x so people could have been running in "production mode" with defaults. Or are you implying that since the new "production mode" stuff is a breaking change, it will force people to set ES_HEAP_SIZE so hence making it a moot point?

@clintongormley

This comment has been minimized.

Copy link
Member

commented May 18, 2016

Or are you implying that since the new "production mode" stuff is a breaking change, it will force people to set ES_HEAP_SIZE so hence making it a moot point?

Precisely

@jasontedor

This comment has been minimized.

Copy link
Member Author

commented May 18, 2016

set ES_HEAP_SIZE

Note that ES_HEAP_SIZE does not exist in >= 5.0.0.

@anhlqn

This comment has been minimized.

Copy link

commented May 20, 2016

Just a note about heap size settings on Windows that could be added to documentation. After installing ES as a service on Windows, if we want to change the heap size, changing the value in jvm.options won't work. Need the change the value after running service.bat manager

@jasontedor

This comment has been minimized.

Copy link
Member Author

commented May 20, 2016

@anhlqn That's always been the case, I'm surprised it's not documented but it definitely does not seem to be. Thanks for the comment, will add.

@jasontedor jasontedor removed the review label May 20, 2016

@jasontedor

This comment has been minimized.

Copy link
Member Author

commented May 26, 2016

@anhlqn I opened #18606.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.