New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch doc_values=true to be the default for non-analyzed string fields #8312

Closed
gibrown opened this Issue Oct 31, 2014 · 11 comments

Comments

Projects
None yet
8 participants
@gibrown
Copy link
Contributor

gibrown commented Oct 31, 2014

We recently started doing a lot of sorting by dates and integer fields on subsets of our 5 billion documents. We are mostly running queries that filter to a few thousand docs and then sorting that subset. Needing to load 5 billion dates into memory for this seemingly common use case make ES feel broken. :)

Heap memory management can be very painful and I think represents a difficult point for new users. Search engines are all about preprocessing data (indexing) so that querying can be fast. In my opinion doc_values accomplishes this better than the current default even if it can be slower in some cases.

FWIW, doc_values is MUCH faster in our use case using 1.3.4.

@bobrik

This comment has been minimized.

Copy link
Contributor

bobrik commented Dec 4, 2014

@gibrown have you seen increased segment memory usage after switching to doc_values? I've seen 2-8x increase (from 32mb to 70-270mb) per index with 120m events.

@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Dec 31, 2014

The JVM heap size is practically limited to 32GB - above this and the JVM can no longer use compressed pointers (resulting in more memory usage for the same data), and garbage collections become slower.

By far the biggest user of the heap for most users is in-memory fielddata, used for sorting, aggregations and scripts. In-memory fielddata is slow to load, as it has to read the whole inverted index and uninvert it. If the fielddata cache fills up, old data is evicted causing heap churn and bad performance (as fielddata is reloaded and evicted again.)

Doc values provide the same function as in-memory fielddata, but the datastructure is written to disk at index time. This results in more disk usage and somewhat slower indexing and mergging (because there is more I/O). Aggregations and sorting are about 20% slower than they are with in-memory fielddata.

The advantages are:

  • less heap usage and faster garbage collections
  • no longer limited by the amount of fielddata that can fit into 32GB of heap - instead the file system caches can make use of all the available RAM
  • fewer latency spikes caused by reloading a large segment into memory

Unfortunately, there is no way to back-fill these values without reindexing. The proposal is to make doc values the default for all fields except analyzed string fields (which are not supported by doc values anyway).

Some users will end up writing much more data than they need, but to be clear: this is a default setting which can be changed as appropriate. It is similar to the fact that we index term positions by default on all analyzed string fields, so as to make phrase queries work out of the box.

Before making this decision, we need to understand the full impact of doc-values-by-default, by running the following tests:

  • ssd vs spinning disk
  • merge throttling enabled/disabled
  • non-string fields vs all not_analyzed fields
  • metric, logging, wikipedia
  • aggs
  • sorting asc and desc
  • multiple aggs and sorting combined

And measuring the following:

  • Node stats after indexing (and always starting from zero)
  • number of segments before optimize
  • index size before/after optimize
  • indexing rate (docs/s)
  • query rate (docs/s)
  • query rate during heavy indexing
  • query rate during heavy indexing with eager loading
  • query rate during heavy indexing with eager global ordinals
  • logs for GC messages and index throttling messages

We need to do this at large scale with a billion documents, on nodes that are properly tuned (eg ES_HEAP, mlockall etc)

Two further proposals:

  • Turn off fielddata loading for analyzed string fields by default. It is too easy to blow up memory usage by sorting or aggregating on an analyzed string field, and it is seldom what the user intended. If the user really does want to run aggregations on an analyzed string field, then it is an easy option to enable on an existing index.
  • Default not_analyzed string fields to have ignore_above set to (eg) 255 characters, as it seldom makes sense to index or write doc values for such large terms
@bharvidixit

This comment has been minimized.

Copy link

bharvidixit commented Mar 13, 2015

@clintongormley what if i enable doc_values=true for the metadata _id field? Will it have some effect? "_id": {
"store": true,
"index": "not_analyzed",
"doc_values": true,
"path": "id"
}

@jpountz

This comment has been minimized.

Copy link
Contributor

jpountz commented Mar 16, 2015

@bharvidixit Actually we are also thinking about having doc values enabled on _uid too. This way, random sorting (which mostly merges a seed with a hash of the _uid to be reproducible) would not need to load fielddata on the _uid field (which takes a lot of memory all the time since this field is unique by definition). And it could also help have consistent pagination by tie-breaking on the _uid instead of the internal lucene doc ids (since they are not the same on all copies of a shard).

@rjernst rjernst removed the discuss label Mar 23, 2015

rjernst added a commit to rjernst/elasticsearch that referenced this issue Mar 27, 2015

Core: Enable doc values by default, when appropriate
Doc values significantly reduced heap usage, which results in faster
GCs. This change makes the default for doc values dynamic: any
field that is indexed but not analyzed now has doc values. This only
affects fields on indexes created with 2.0+.

closes elastic#8312
closes elastic#10209
@gibrown

This comment has been minimized.

Copy link
Contributor Author

gibrown commented Mar 27, 2015

Awesome, Thanks!

@bobrik

This comment has been minimized.

Copy link
Contributor

bobrik commented Mar 27, 2015

👍

@jknewman3

This comment has been minimized.

Copy link

jknewman3 commented Apr 1, 2015

Is it possible to set eager loading, or something like that, for doc_values?
It would be helpful for us to always have the most recently added data in cache.
And, yeah, warmers could do it for us. Just wondering if eager applies only to JVM.

@rjernst

This comment has been minimized.

Copy link
Member

rjernst commented Apr 2, 2015

@jknewman3 See discussions on #8693

@jknewman3

This comment has been minimized.

Copy link

jknewman3 commented Apr 2, 2015

Ah, that helps a lot. Thanks!

@bolee

This comment has been minimized.

Copy link

bolee commented Aug 6, 2015

how to set doc_value=true, via api to set every field?

@clintongormley

This comment has been minimized.

Copy link
Member

clintongormley commented Aug 6, 2015

@bolee please ask questions like that in the forum: https://discuss.elastic.co/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment