Frozen Indices

[s1monw]

Frozen indices are intended to allow much higher ratios of disk storage to heap, at the expense of search latency. The idea is to keep frozen indices in an unloaded state (ie not loading lucene data-structures into heap) until they are searched. When a search targets frozen indices, each index would be searched sequentially (instead of in parallel), and each index would be loaded, searched, then unloaded again. Frozen indices would be replicated, unlike closed indices today. In-fact frozen indices are more like open indices with metadata in memory and data fully on disk until it's needed to be loaded

Frozen indices will be available in the default distribution but not in the pure OSS distribution.

In oder to implement this, the feature is broken down to the following steps:

• Add the ability to force searches to go through a threadpool that has a limited amount of threads to ensure shards data is materialized sequentially or with very limited concurrency. This will allow frozen indices to be marked as index.search.throttled that will force searches as well as other data access like _get or _explain through a dedicated search_throttled threadpool which has only 1 thread by default. (Introduce a search_throttled threadpool #33732)
• Prevent throttled indices to be included in _search and _msearch by default. This will prevent frozen indices to be included in wildcard or alias searches by default but allows opt-in to searching these indices. At the same time these indices are treated like ordinary indices in all other APIs (Prevent throttled indices to be searched through wildcards by default #34354)
• Add a frozen engine that maintains static stats and opens an index reader on demand. The searcher / reader returned form this engine will be able to reset itself after it search phase to release resources and open up a new searcher / reader from the same commit point on subsequent search phases like fetch or scan / scroll (Add a frozen engine implementation #34357)
• Add an Java API that allows to open a closed index as frozen. This is the simplest way to introduce frozen indices, further improvements on how we go from Open to Frozen can be done as followups if necessary. As a start an index must got through Closed in order to be frozen or unfrozen. (Add a java level freeze/unfreeze API #35353)
• Add an optimization that returns a special index reader that allows range queries on date fields to rewrite in can_match phase in order to exclude frozen indices in can_match without opening up their index reader. (Allow efficient can_match phases on frozen indices #35431)
• Add a REST API and Documentation for frozen indices and how to freeze and index (Add a _freeze / _unfreeze API #35592)
• Add high-level REST client support for _freeze / _unfreeze(Add high-level REST client API for _freeze and _unfreeze #35723)

[s1monw]

@debadair where do you see the documentation go for this feature?

[elasticmachine]

Pinging @elastic/es-distributed

[s1monw]

I thought about some future work on this and wanted to put it here for awareness and potential discussion:

• The current design would open a reader before we check the cache for hits and even if we cache we would invalidate the cache after each search phase. I wonder if we can trick the lazy reader design into having it's own cache key helper that we can trick into surviving close calls of the actual reader. That way we might be able to allow request cache hits on frozen indices without actually opening the reader

• should we allow forcemerge on a frozen index? This might allow us to freeze first and the select a new codec to get best compression. It’s relatively straight forward to do with our infrastructure and an IW that is not pulling NRT readers. And we have limited threads doing this anyway in the force merge threadpool. That's totally something we can do way later.

[cdahlqvist]

When benchmarking frozen indices to get an idea of the overhead and added latency it adds I can see two common scenarios:

1) Dedicated Cold nodes

Here all indices will be frozen, and due to the low heap usage it may be possible to hold a lot of data with a small heap. In order to efficiently use the available resources and allow parallel processing of frozen indices, I assume it may make sense to set the number of threads used for frozen indices processing to the number of CPU cores on the host. This naturally assumes there is enough heap available to support this.

2) Warm nodes

Frozen indices could also be used to make Warm nodes able to handle larger amounts of data. Here the most recent and generally most queried data would be open, while older indices that are not queried frequently could be frozen. The ratio of frozen to open indices will naturally vary from use-case to use-case, but for the benchmark a 50% ratio might be a good starting point. In this scenario, a single thread used to handle frozen indices is probably appropriate.

Does this make sense?

In addition to benchmarking these two scenarios, I would also like to benchmark using indices that have and have not been forcemerged down to a single segment since restoration time here may vary. If we add a reference benchmark for each scenario with all indices open, we get a total of 8 benchmark runs.

Benchmark set-up

Test data would be generated using the rally-eventdata-track. A single index with 2 primary shards and around 50GB in size would probably be reasonably realistic. Snapshots would be created before and after forcemerging it down to a single segment. This index can then be restored and renamed multiple times to reach a reasonable data volume (~1TB?). Querying would simulate the standard Kibana dashboards available in the rally-eventdata-track. As Kibana dashboards can require a good amount of processing, I would probably run querying in a single thread.

As querying using Rally is quite light, we should be able to run this on a single host. A AWS EC2 d2.xlarge instance might be a good choice as it has enough disk space and used the type of slower storage we often see for these node types.

[s1monw]

Here all indices will be frozen, and due to the low heap usage it may be possible to hold a lot of data with a small heap. In order to efficiently use the available resources and allow parallel processing of frozen indices, I assume it may make sense to set the number of threads used for frozen indices processing to the number of CPU cores on the host. This naturally assumes there is enough heap available to support this.

I would like to keep the thread-settings for this to the default. We can still test this but that is for a later iteration.

Frozen indices could also be used to make Warm nodes able to handle larger amounts of data. Here the most recent and generally most queried data would be open, while older indices that are not queried frequently could be frozen. The ratio of frozen to open indices will naturally vary from use-case to use-case, but for the benchmark a 50% ratio might be a good starting point. In this scenario, a single thread used to handle frozen indices is probably appropriate.

In general I think this scenario is rare and benchmarking will be tricky. I don't want to give the impression that this is a recommended setup so I'd rather not advertise it too much.

[s1monw]

This issue is basically done. I am keeping this open in order to publish numbers from benchmarks in the near future.

[cdahlqvist]

I have now run a first set of benchmarks for frozen indices where the number of indices in the cluster gradually increases from 1 to 18. Each index consists of 2 shards and is ~50GB in size and was generated and queried using the rally-eventdata-track. This was run against a build with the build hash 447e5d2.

Querying was done using two types of simulated Kibana queries:

• Kibana Discover query: Single histogram aggregation and 500 documents being retrieved
• Content Issues Dashboard query: Small dashboard containing 5 aggregations, aggregating over a subset of the data.

For indices forcemerged down to a single segment, two different heap sizes were used when querying frozen indices: 1GB and 15GB. For the larger heap, a reference run with all indices open was performed at every step. Each run consists of 20 queries with a varying time interval, and 50% and 100% percentile service time values are shown.

The results are shown below in the form of Kibana screenshots from the Rally metrics store.

Discover query, 15GB Heap, Forcemerged indices, all indices open and not frozen

Discover query, 1GB/15GB Heap, Forcemerged indices, all indices frozen

Content Issues Dashboard query, 15GB Heap, Forcemerged indices, all indices open and not frozen

Content Issues Dashboard query, 1GB/15GB Heap, Forcemerged indices, all indices frozen

[s1monw]

thanks you @cdahlqvist great results. I will now close this issue. thanks everyone involved.

[octocat-mona]

should we allow forcemerge on a frozen index?

@s1monw Maybe I missed the answer in this thread, but this is/won't be allowed?
Because the docs mention that a _forcemerge should only be called on read-only indices it made sense to me to use _freeze because it does exactly this: make it read-only.

But it took me a while to figure out that _forcemerge doesn't do anything on a frozen index and doesn't fail either..